multi-arch-builders: use splunk to monitor and send logs #894

marmijo · 2023-07-21T22:06:11Z

Add a splunk forwarder sidecar container to the RHCOS multiarch builders that will send journald logs to a central splunk server.

commit ec961fe76eeaf518535d36a221287cdb133ea5a7
Author: Michael Armijo <marmijo@redhat.com>
Date:   Mon Jul 24 02:20:39 2023 -0400

    multi-arch-builders: add additional RHCOS specific configs
    
    Add RHCOS specific configs for `aarch64` and `ppc64le` that merges
    the newly created splunk cofig with the arch specific configs.

commit 309e991f89af0b97875a34a9ad1a3462641f1af1
Author: Michael Armijo <marmijo@redhat.com>
Date:   Mon Jul 24 02:19:11 2023 -0400

    multi-arch-builders: include splunk config in `s390x` RHCOS builder
    
    Include the newly created splunk config when creating the `s390x`
    remote builder.

commit bb2e704cfa61607441226fe863d6cd0aa818b55f
Author: Michael Armijo <marmijo@redhat.com>
Date:   Fri Jul 21 17:51:48 2023 -0400

    multi-arch-builders: add splunk butane config
    
    Add a butane config to the multi-arch builders to include when creating the
    RHCOS builders. This butane config will build and start a splunk container
    to send logs to a central RH splunk server.
    
    See: https://issues.redhat.com/browse/COS-2131

jschintag · 2023-07-26T15:28:45Z

multi-arch-builders/coreos-builder-splunk.bu

+          ExecStartPre=nm-online --timeout=30
+          # use `--arch=x86_64` because the RPM doesnt exist for aarch64. ppc64le and s390x RPMs dont work for journald input.
+          ExecStart=-podman build --pull-always --cache-ttl=480h --arch=x86_64 -t localhost/splunkforwarder:latest https://gitlab.corp.redhat.com/paas/cicd.git#main:itpaas-util-images/splunkforwarder
+          ExecStart=-podman build --build-arg project=${SPLUNK_PROJECT} --cache-ttl=480h --from localhost/splunkforwarder:latest -t localhost/splunkforwarder-rhcos https://gitlab.corp.redhat.com/paas/docker-paas.git#master:splunk


Is there no way to build the container and just pull it from the registry here?
Building the container each time on reboot sounds failure prone and unnecessary.

Also, if using pre-built container images, it would be possible to poll for updates regulary, instead of relying on reboots for updates. Not sure how often that would happen, but we might want to have those run unattended, as that is most of the reason we are using fcos for the pipeline.

@dustymabe and I talked about this. If we push it to a registry, we most likely need it locked down somewhere. That would mean configuring extra credentials on the builders when we install. Building the container each time also reduces the burden of maintaining a separate container build workflow.

If you already talked through the implications of building it and decided it is the better option i am fine with that.

What about the automated updates? I guess right now it would happen each time fcos is updated (rebooted) right? Is that frequent enough?

@jschintag if the container fails to build on any particular boot then we'll just continue to use the previous container image that was built on a previous boot.

If we do more than one reboot quickly the --cache-ttl=480h will essentially make the next reboot builds a no-op.

For updates, yeah. The container would get rebuilt every two weeks when FCOS goes down for automated updates.

mike-nguyen · 2023-07-27T01:40:54Z

multi-arch-builders/coreos-builder-splunk.bu

+          Environment=GIT_SSL_NO_VERIFY=1
+          ExecStartPre=nm-online --timeout=30
+          # use `--arch=x86_64` because the RPM doesnt exist for aarch64. ppc64le and s390x RPMs dont work for journald input.
+          ExecStart=-podman build --pull-always --cache-ttl=480h --arch=x86_64 -t localhost/splunkforwarder:latest https://gitlab.corp.redhat.com/paas/cicd.git#main:itpaas-util-images/splunkforwarder


Is it also possible to hide the internal gitlab address?

We could maybe have a generic internal HTTP Link that contains the address, but it would of course introduce new dependencies

Another option could be to replace the address with ${ITPAAS_SPLUNK_URL}. There are two other env variables in this file that will have to be resolved by creating the ignition config like:

envsubst < coreos-builder-splunk.bu | butane --pretty --strict > coreos-builder-splunk.ign

I'm creating an internal readme explaining the splunk container and how to install it, including which variables to set during the installation of the RHCOS remote builders. I could have the URL documented there as well.

yeah, if we want to hide it then envsubst is probably the way to go.

dustymabe · 2023-07-27T18:20:12Z

multi-arch-builders/coreos-builder-splunk.bu

+          Volume=/var/log/journal:/var/log/journal:ro
+          Volume=/run/log/journal:/run/log/journal:ro
+          Volume=/etc/machine-id:/etc/machine-id:ro
+          Volume=/var/home/core:/var/log/journald:ro


this seems a bit odd. This is a container running under the splunk user but we mount the core user's home directory into the container under /var/log/journald?

Oops, carried over from debugging before we officially got journald logs into splunk. I'll remove this

dustymabe · 2023-07-27T18:21:33Z

multi-arch-builders/coreos-builder-splunk.bu

+          Volume=/run/log/journal:/run/log/journal:ro
+          Volume=/etc/machine-id:/etc/machine-id:ro
+          Volume=/var/home/core:/var/log/journald:ro
+          Network=host


Can you explain why Network=host is needed? Maybe add a comment?

dustymabe · 2023-07-27T18:22:34Z

multi-arch-builders/coreos-builder-splunk.bu

+          ContainerName=rhcos-multiarch-splunk
+          Image=localhost/splunkforwarder-rhcos:latest
+          RunInit=true
+          User=root


add comment maybe?

though now I have a question.. this means that it runs as "root" in your user namespace (i.e. as UID=0 inside the container which is equivalent to the splunk user on the host), right. It's not running as actual "root" on the host?

Correct, we're running the container as the splunk user on the host, not the root user

dustymabe · 2023-07-27T18:23:16Z

multi-arch-builders/coreos-builder-splunk.bu

+          [Container]
+          ContainerName=rhcos-multiarch-splunk
+          Image=localhost/splunkforwarder-rhcos:latest
+          RunInit=true


maybe add a comment here too if we have the context.

dustymabe · 2023-07-27T18:24:16Z

multi-arch-builders/coreos-builder-splunk.bu

+          Volume=/etc/machine-id:/etc/machine-id:ro
+          Volume=/var/home/core:/var/log/journald:ro
+          Network=host
+          PodmanArgs=--ipc=host --group-add keep-groups --hostname=${SPLUNK_HOSTNAME}


--ipc=host is needed for journalctl to talk to the journal on the host?

--group-add keep-groups is needed to pick up the wheel and systemd-journal groups from the splunk user?

I added a comment explaining why 3 additional options are needed

dustymabe · 2023-07-27T18:30:34Z

multi-arch-builders/coreos-builder-splunk.bu

+          Environment=GIT_SSL_NO_VERIFY=1
+          ExecStartPre=nm-online --timeout=30
+          # use `--arch=x86_64` because the RPM doesnt exist for aarch64. ppc64le and s390x RPMs dont work for journald input.
+          ExecStart=-podman build --pull-always --cache-ttl=480h --arch=x86_64 -t localhost/splunkforwarder:latest https://gitlab.corp.redhat.com/paas/cicd.git#main:itpaas-util-images/splunkforwarder


yeah, if we want to hide it then envsubst is probably the way to go.

dustymabe · 2023-07-27T18:32:34Z

multi-arch-builders/coreos-builder-splunk.bu

+          ExecStartPre=nm-online --timeout=30
+          # use `--arch=x86_64` because the RPM doesnt exist for aarch64. ppc64le and s390x RPMs dont work for journald input.
+          ExecStart=-podman build --pull-always --cache-ttl=480h --arch=x86_64 -t localhost/splunkforwarder:latest https://gitlab.corp.redhat.com/paas/cicd.git#main:itpaas-util-images/splunkforwarder
+          ExecStart=-podman build --build-arg project=${SPLUNK_PROJECT} --cache-ttl=480h --from localhost/splunkforwarder:latest -t localhost/splunkforwarder-rhcos https://gitlab.corp.redhat.com/paas/docker-paas.git#master:splunk


@jschintag if the container fails to build on any particular boot then we'll just continue to use the previous container image that was built on a previous boot.

If we do more than one reboot quickly the --cache-ttl=480h will essentially make the next reboot builds a no-op.

For updates, yeah. The container would get rebuilt every two weeks when FCOS goes down for automated updates.

dustymabe · 2023-07-27T18:35:14Z

multi-arch-builders/coreos-builder-splunk.bu

+          # Figure out soon if we can get the Red Hat Certs onto the hosts
+          Environment=GIT_SSL_NO_VERIFY=1


I still need to look into this.

multi-arch-builders/coreos-builder-splunk.bu

multi-arch-builders/coreos-ppc64le-builder-512e.bu

dustymabe

Some comments.. We're getting really close, but I still need to investigate GIT_SSL_NO_VERIFY too.

Do you mind also rebasing on top of latest main branch?

dustymabe · 2023-07-28T20:33:18Z

multi-arch-builders/builder-splunk.bu

+          # use the container host's network to properly establish
+          # a connection with the splunk server
+          Network=host


Even with the comment I still don't quite understand why this is needed maybe we can get toghether and I can understand more?

Sure we can chat more about it. I got this information from the ITPAAS internal documentation and code. The splunk team also advised me to use this option. I tried to run the container without it and it wouldn't connect to to the server.

interesting. would definitely be nice to know why - maybe some issues with NAT?

dustymabe · 2023-07-28T20:35:09Z

multi-arch-builders/builder-splunk.bu

+          # use `--ipc=host` to use the container host's IPC namespace
+          # to properly establish a connection with the splunk server


usually ipc is for processes to communicate with each other on the same host IIUC. Is there message passing being done back and forth between journalctl running in the container and some daemon on the host? or something else going on?

I would imagine it has something to do with the splunk process itself needing to communicate with the server. I was advised by the splunk team to use it this way. We can chat more about this 1:1.

Hmm. I tried googling around a bit but couldn't find anything. Can you confirm that if you don't have this setting it doesn't work?

Yes, I tried it earlier today, and again just now to make sure. It doesn't work without --ipc=host. I can see a connection with splunk, but no logs are being sent to the server.

ok maybe just add a "we don't know why this is needed but it doesn't work without it" to the comment.

multi-arch-builders/builder-splunk.bu

dustymabe · 2023-07-28T20:37:29Z

multi-arch-builders/builder-splunk.bu

+          # The --arch=x86_64 works here because we have `qemu-user-static-x86`
+          # installed on non-x86_64 FCOS streams.
+          ExecStart=-podman build --pull-always --cache-ttl=480h --arch=x86_64 -t localhost/splunkforwarder:latest ${ITPAAS_SPLUNK_REPO}
+          ExecStart=-podman build --build-arg project=${DEPLOYMENT_CLIENT} --cache-ttl=480h --from localhost/splunkforwarder:latest -t localhost/splunkforwarder-rhcos ${SPLUNK_SIDECAR_REPO}


feels like we don't really need to obfuscate the value of DEPLOYMENT_CLIENT do we?

I'm not sure about this. It is a piece of the internal configuration that's used to establish a connection with the server.

right, but it's just a string. I don't think there's really any need to hide it, but I'm fine with leaving it as it is.

marmijo · 2023-07-28T21:15:31Z

Do you mind also rebasing on top of latest main branch?

Done!

Add a butane config to the multi-arch builders to include when creating the RHCOS builders. This butane config will build and start a splunk container to send logs to a central RH splunk server. See: https://issues.redhat.com/browse/COS-2131

Include the newly created splunk config when creating the `s390x` remote builder.

Add RHCOS specific configs for `aarch64` and `ppc64le` that merges the newly created splunk cofig with the arch specific configs.

dustymabe

LGTM

marmijo changed the title ~~WIP remote-builders: use splunk to monitor and send logs~~ WIP: remote-builders: use splunk to monitor and send logs Jul 21, 2023

marmijo marked this pull request as draft July 21, 2023 22:07

marmijo force-pushed the add_splunk_config branch from 7e48a5f to 881a158 Compare July 24, 2023 06:34

marmijo changed the title ~~WIP: remote-builders: use splunk to monitor and send logs~~ multi-arch-builders: use splunk to monitor and send logs Jul 24, 2023

marmijo force-pushed the add_splunk_config branch 2 times, most recently from d678c41 to da49276 Compare July 25, 2023 22:24

jschintag reviewed Jul 26, 2023

View reviewed changes

mike-nguyen reviewed Jul 27, 2023

View reviewed changes

dustymabe reviewed Jul 27, 2023

View reviewed changes

marmijo force-pushed the add_splunk_config branch from da49276 to 4007a6a Compare July 28, 2023 20:24

marmijo marked this pull request as ready for review July 28, 2023 20:24

dustymabe reviewed Jul 28, 2023

View reviewed changes

marmijo force-pushed the add_splunk_config branch from 4007a6a to 048f211 Compare July 28, 2023 21:14

marmijo added 3 commits July 28, 2023 20:55

multi-arch-builders: include splunk config in s390x RHCOS builder

309e991

Include the newly created splunk config when creating the `s390x` remote builder.

multi-arch-builders: add additional RHCOS specific configs

ec961fe

Add RHCOS specific configs for `aarch64` and `ppc64le` that merges the newly created splunk cofig with the arch specific configs.

marmijo force-pushed the add_splunk_config branch from 048f211 to ec961fe Compare July 29, 2023 03:42

dustymabe approved these changes Jul 29, 2023

View reviewed changes

marmijo merged commit 4b8d009 into coreos:main Jul 30, 2023
2 checks passed

marmijo deleted the add_splunk_config branch September 1, 2023 17:34

		# Figure out soon if we can get the Red Hat Certs onto the hosts
		Environment=GIT_SSL_NO_VERIFY=1

		# use `--ipc=host` to use the container host's IPC namespace
		# to properly establish a connection with the splunk server

multi-arch-builders: use splunk to monitor and send logs #894

multi-arch-builders: use splunk to monitor and send logs #894

Conversation

marmijo commented Jul 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marmijo Jul 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustymabe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marmijo commented Jul 28, 2023

dustymabe left a comment

Choose a reason for hiding this comment

marmijo commented Jul 21, 2023 •

edited

Loading

marmijo Jul 27, 2023 •

edited

Loading