Add k8s manifests to deploy stack along with Agent

[ChrsMark]

This PR adds Kubernetes manifests to be used when Agent is needed to run in k8s along with Elastic Stack.

Quick start:
Having a k8s cluster running run kubectl apply -f snapshot.yml and after the stack is ready enrol the Agent with kubectl apply -f agent.yml

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #89 updated]

• Start Time: 2020-06-26T07:36:31.896+0000

• Duration: 3 min 56 sec

[mtojek]

Wow, didn't expect this one, nice work! @ruflin apparently, it's a new kind of fleet agent usage :) Could you please add a README file with some commands to apply manifests?

Also, I'm sure if I see a reason to put everything on the Kubernetes cluster (ES, Kibana, etc.). I thought the Elastic cluster could live aside and the agent monitor the Kubernetes API.

[exekias]

Thank you for working on this! Are these minded for development or as a reference for users? I think it would worth adding these to agent docs, same way we have a Running Beats on Kubernetes section

[ChrsMark]

Wow, didn't expect this one, nice work! @ruflin apparently, it's a new kind of fleet agent usage :) Could you please add a README file with some commands to apply manifests?

@mtojek I added the READMe after playing a little bit and resolving a couple of issues.
Now this env seems to be quite stable and enrols the Agent with success providing system metrics in Kibana.

Also, I'm sure if I see a reason to put everything on the Kubernetes cluster (ES, Kibana, etc.). I thought the Elastic cluster could live aside and the agent monitor the Kubernetes API.

Well, different Deployment options can be used for sure. ES, Kibana and Registry can be deployed everywhere as soon as they are accessible from Agent. In order to reproduce this env locally deploying everything on k8s seems to be the easiest way to avoid networking/accessibiltiy issues etc.

Reasoning for running Agent on k8s:
In order to enable state_* metricsets of Kubernetes module one needs access to kube_state_metrics service which runs within k8s cluster. Currently it is suggested to deploy Metricbeat as k8s Deployment (with 1 Pod) in order to collect these cluster-wide metrics. Of course this can be achieved by having Agent running outside of the cluster and querying kube_state_metrics if the service is actually exposed to the outside world. Then we will need access to the k8s API too in order to leverage add_kubernetes_metadata. Then issues regarding certificates, accessibility etc may occur so we end up that deploying Agent/Metricbeat within the k8s cluster is the best option to go with when it comes to the Kubernetes module (sorry, package 😅 )(https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat/metricbeat-deployment-configmap.yaml#L50).

To add more to the above statement, pod, node, container etc metricsets require access to the Kubelet's API of each node. This seems to be impossible to happen from outside of the cluster. Currently we enable these Metricsets on Deamonset Pods running on each of the k8s nodes (https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat/metricbeat-daemonset-configmap.yaml#L79).

In order to combine the above 2 different deployment strategies we could suggest that the users create 2 different configs in Ingest Manager one for the cluster-wide metrics and one for the metrics collected from each node. Then the singleton agent will be enrolled using the cluster-wide config while the Agents from the Daemonset Pods will be enrolled using the node-level config.
This will provide k8s native experience to the users that aim to use our Kubernetes package.
This is a solution we found with @ycombinator after brainstorming around this story. Please feel free to provide feedback 🙂 .

Screenshot regarding the 2 configs:

Then deploying the package in the 2 different agent groups:

Functional overview dashboard:

[ChrsMark]

Thank you for working on this! Are these minded for development or as a reference for users? I think it would worth adding these to agent docs, same way we have a Running Beats on Kubernetes section

Well, for now it is aimed to be used for development reasons as part of #70. The goal is to use these in order to migrate the Kubernetes module/package. Then for sure we can provide the Running Agent on Kubernetes to the public.

Note that Agent's manifest is still under "development" since I will need to add RBACs etc in order to make it capable to run Kubernetes module.

[mtojek]

To be honest, I would put/quote your comment (#89 (comment)) in the README file ;) By looking at all required permissions, limitations to run the agent outside of the cluster, I agree with you that's nearly impossible or at least not reasonable.

I wonder if it shouldn't be responsibility of the agent team to support the k8s runtime. WDYT @ruflin?

@ChrsMark do you need an approval or do you prefer to add more changes here?

[ChrsMark]

To be honest, I would put/quote your comment (#89 (comment)) in the README file ;)

When we have something stable we can include the reasoning/explanation of deployment approach to the respective docs, READMEs etc.

@ChrsMark do you need an approval or do you prefer to add more changes here?

Not mergable yet.

[mtojek]

Not mergable yet.

If it doesn't introduce any complications, please convert it to draft.

[ruflin]

As soon as we make this available to our customers, the agent team should probably own this. But at the moment this is focused on testing and local usage so the integrations repository sounds like a good place to kick this off.

[ChrsMark]

Note that in order to release the package we most probably need to provide the k8s manifests to the users too, along with usage docs.

[ChrsMark]

This one seems to be ok to be merged in this repo for testing/development purposes. @exekias if you have the time to check for anything that could be missing, it would be great.

Opening this for review for now.

[mtojek]

Thanks!

Could you please make sure, that's aligned to latest updates to testing/environment? There was an issue with missing "encryptionKey".

[ChrsMark]

Updated the manifests according to the latest updates of testing/env.