*: Data collection from FCOS machines #31

zonggen · 2019-09-18T20:33:28Z

Source of truth for each part of data collected:

Minimal:
- platform: read_file /proc/cmdline
- current_os_version: read_output rpm-ostree status --json
- original_os_version: read_file /.coreos-aleph-version.json
- instance_type: read_file /run/metadata/afterburn
Full:
- hardware:
  - disk: read_output lsblk --fs --json
  - cpu: read_output lscpu --json
  - memory: read_output lsmem --json
- network: read_output nmcli device show
- container_runtime:
  - podman:
    - check if running by pgrep podman
    - count running container by pgrep conmon | wc -l
  - docker:
    - check if running by pgrep dockerd
    - count running container by pgrep containerd-shim | wc -l
  - cri-o:
    - check if running by pgrep crio
    - count running container by pgrep crictl | wc -l
  - system-nspawn:
    - check if running and counting containers by pgrep systemd-nspawn

Signed-off-by: Allen Bai, abai@redhat.com

zonggen · 2019-09-18T20:40:15Z

Currently in main.rs the collected ignition platform id will be printed to the console, which will be updated later to be bundled with other data and sent to the remote endpoint.

zonggen · 2019-09-18T20:41:13Z

Also, this PR will be updated to resolve #30 (comment)

src/main.rs

src/identity/platform.rs

src/identity/mod.rs

Collects ignition platform id from /proc/cmdline in the varriable `ignition.platform.id=` and stores the collected data inside the `identity::Identity` struct. Also adds different grains of data collection level: minimum and full, depending on the passed toml configuration. Signed-off-by: Allen Bai, abai@redhat.com

Changes the use of term "metrics", deduplicates match pattern and uses maplit::hashmap! macro.

zonggen · 2019-09-19T21:20:03Z

UPDATE:

Added the module os-release.rs under identity to extract OS version from /etc/os-release

Note: Most portion of code is the same as platform.rs so in the future, towards the end of this PR should de-duplicate this part of code into a new utils helper module.

Picks up CoreOS alpha version information from `/.coreos-aleph-version.json`. Signed-off-by: Allen Bai, abai@redhat.com

jlebon

Looks sane from a quick scan!

src/identity/mod.rs

src/identity/os_release.rs

src/full: add hardware data collection Signed-off-by: Allen Bai <abai@redhat.com>

Signed-off-by: Allen Bai <abai@redhat.com>

zonggen · 2019-10-15T21:32:57Z

STATUS:
Currently Travis CI is failing since it is using Rust docker image which is based on Debian and it does not ship with lsmem, and lscpu does not contain an --json option.
To test with more production-like environment, i.e. FCOS, it might be better to switch to UBI (https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image). Will be updating with changes to see if it works.

EDIT:

Updated .travis.tml to use build a container using Dockerfile and test the code inside the UBI container

.travis.yml: test inside ubi container instead of ubuntu vm Allows "yes"/"no" values for `removable` field of `lsmem --json`, because of different implementations, the field type might be different. However, in an FCOS machine this will be true/false. This change is mainly to accommodate testing environment. Signed-off-by: Allen Bai <abai@redhat.com>

Signed-off-by: Allen Bai <abai@redhat.com> agent/full: only collect hw info on bare metals Signed-off-by: Allen Bai <abai@redhat.com> full/network: collect network info with `nmcli` Collects network data with format `[key[\s]+value\n]+`, to see example run `nmcli device show`. Parse the data as key:value pairs stored in a HashMap. Signed-off-by: Allen Bai <abai@redhat.com>

Signed-off-by: Allen Bai <abai@redhat.com>

zonggen · 2019-10-21T16:16:29Z

STATUS:

Since collecting service is meant to running in the background and users should not expect visible error information, the program should not crash/return error when one of the commands returns non-zero and instead should suppress the error and collect whatever is available at the moment.
Could use a log file to dump the error message

tests: add tests to print minimum and full agent with test config file Signed-off-by: Allen Bai <abai@redhat.com>

Signed-off-by: Allen Bai <abai@redhat.com>

zonggen · 2019-10-22T16:16:54Z

STATUS:

Client side data collection is done. though the send_data() function is not doing anything other than printing the collected data in Json format. This needs to be updated after we have an actual endpoint.
Persistent state so that pinger could send data periodically with some fixed interval will be added through another PR to reduce the size of this PR

Overall, ready for review!

Dockerfile: install dependency, openssl-devel Signed-off-by: Allen Bai <abai@redhat.com>

Since reporting will happen daily or monthly, we are keeping track of time intervals between reports. Two threads are spawned for tracking daily and monthly reporting. Daily thread will check timestamp every 12 hours, otherwise put to sleep. Monthly thread will check every 15 days, otherwise put to sleep. Signed-off-by: Allen Bai <abai@redhat.com>

zonggen · 2019-10-24T20:47:03Z

STATUS:

Built on top of systemd: add persistent state and report with different intervals #35, in the main function, to create a Daemon process, one process is forked and parent process is immediately terminated. Two threads are then spawned by child process to track the time for daily and monthly reports.
The two threads are put into sleep to avoid busy waiting, and only checks whether it needs to send new report every 12 hours / 15 days for daily and monthly reports correspondingly.

zonggen · 2019-10-28T13:30:51Z

cc @lucab @bgilbert

…time Previously we are using individual container runtimes to extract container information, e.g. calling `podman container ls` to count running containers. It would create two problems: - podman failed when trying to create `/.config/containers` since root dir is read-only and dynamic user does not have its own home dir - successful calls do not make sense either since counting running containers under the dynamic user is not useful Hence, switch to count containers by calling `pgrep` and count the running processes of each container runtime. Signed-off-by: Allen Bai <abai@redhat.com>

Previously used `StateDirectory=` to persist pinger specific data across bootups. It turns out that systemd does not allow sharing state directory across different dynamic users. An example would be: ``` $ systemd-run --pty --property=DynamicUser=yes --property=StateDirectory=wuff /bin/sh $ systemd-run --pty --property=DynamicUser=yes --property=StateDirectory=wuff /bin/sh ``` Reference: http://0pointer.net/blog/dynamic-users-with-systemd.html Signed-off-by: Allen Bai <abai@redhat.com>

ashcrow · 2021-03-18T13:53:39Z

Is this something we want to continue pursuing?

bgilbert · 2021-03-18T19:13:23Z

Maybe not; xref coreos/fedora-coreos-tracker#770.

zonggen changed the title ~~src/identity: Add a module that collects platform id~~ [WIP] src/identity: Add a module that collects platform id Sep 18, 2019

zonggen changed the title ~~[WIP] src/identity: Add a module that collects platform id~~ [WIP] *: Metrics collection from FCOS machines Sep 18, 2019

lucab reviewed Sep 18, 2019

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

src/identity/platform.rs Outdated Show resolved Hide resolved

src/identity/mod.rs Outdated Show resolved Hide resolved

src/identity/mod.rs Outdated Show resolved Hide resolved

Allen Bai added 2 commits September 19, 2019 10:14

*: Update according to reviews

1d92353

Changes the use of term "metrics", deduplicates match pattern and uses maplit::hashmap! macro.

zonggen force-pushed the collect-metrics branch from 9f28b8e to 1d92353 Compare September 19, 2019 15:30

identity/os-release: Add a module to abstract os version

2c66d08

identity/os_release.rs: Collect CoreOS alpha version

140e9e0

Picks up CoreOS alpha version information from `/.coreos-aleph-version.json`. Signed-off-by: Allen Bai, abai@redhat.com

jlebon reviewed Sep 25, 2019

View reviewed changes

src/identity/mod.rs Outdated Show resolved Hide resolved

src/identity/os_release.rs Outdated Show resolved Hide resolved

src/providers: Extract cloud instance type in mod providers

9f95f1d

zonggen force-pushed the collect-metrics branch from bcde4ce to 9f95f1d Compare September 26, 2019 16:18

Allen Bai added 2 commits September 30, 2019 09:54

os_release.rs: Change name alpha to aleph

ade473b

rpm-ostree: Add rpm-ostree module for os version extraction

da6a36c

zonggen force-pushed the collect-metrics branch 3 times, most recently from 97f3a1d to b3a806b Compare October 9, 2019 20:09

*: cleanup redundant code imported from Afterburn

b35c9a1

zonggen force-pushed the collect-metrics branch from b3a806b to b35c9a1 Compare October 9, 2019 20:31

Allen Bai added 3 commits October 15, 2019 11:28

src/agent: merge minimal and full under the same module, agent

c50cb84

src/full: add hardware data collection Signed-off-by: Allen Bai <abai@redhat.com>

minimal/mock_tests: add mock test for minimal agent

7a05d4f

Signed-off-by: Allen Bai <abai@redhat.com>

full/mock_tests: add mock tests for full agent

4013ebf

Signed-off-by: Allen Bai <abai@redhat.com>

zonggen force-pushed the collect-metrics branch 2 times, most recently from 96eb3d1 to 01b0a44 Compare October 16, 2019 17:21

zonggen force-pushed the collect-metrics branch from 01b0a44 to d209da0 Compare October 16, 2019 18:17

zonggen changed the title ~~[WIP] *: Metrics collection from FCOS machines~~ [WIP] *: Data collection from FCOS machines Oct 16, 2019

Allen Bai added 2 commits October 16, 2019 17:20

src/full: add module for extracting container runtime info

6a5e8f1

Signed-off-by: Allen Bai <abai@redhat.com>

agent/minimum: set default value for identity instead of returning error

ec203b0

tests: add tests to print minimum and full agent with test config file Signed-off-by: Allen Bai <abai@redhat.com>

zonggen force-pushed the collect-metrics branch from 05db91d to ec203b0 Compare October 21, 2019 19:11

main.rs: serialize the Agent struct and POST as json

e650f00

Signed-off-by: Allen Bai <abai@redhat.com>

zonggen force-pushed the collect-metrics branch from 5d8917e to e650f00 Compare October 22, 2019 15:57

zonggen changed the title ~~[WIP] *: Data collection from FCOS machines~~ *: Data collection from FCOS machines Oct 22, 2019

*: add some documentation for each module

5267871

Dockerfile: install dependency, openssl-devel Signed-off-by: Allen Bai <abai@redhat.com>

zonggen force-pushed the collect-metrics branch from 8732dfc to 5267871 Compare October 22, 2019 16:39

zonggen requested review from lucab and jlebon October 31, 2019 17:02

zonggen force-pushed the collect-metrics branch from 0076acf to a0ab83a Compare November 21, 2019 16:35

zonggen force-pushed the collect-metrics branch from a0ab83a to 4387efa Compare November 21, 2019 16:36

rfairley self-requested a review June 4, 2020 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Data collection from FCOS machines #31

*: Data collection from FCOS machines #31

zonggen commented Sep 18, 2019 •

edited

Loading

zonggen commented Sep 18, 2019

zonggen commented Sep 18, 2019

zonggen commented Sep 19, 2019 •

edited

Loading

jlebon left a comment

zonggen commented Oct 15, 2019 •

edited

Loading

zonggen commented Oct 21, 2019

zonggen commented Oct 22, 2019

zonggen commented Oct 24, 2019

zonggen commented Oct 28, 2019

ashcrow commented Mar 18, 2021

bgilbert commented Mar 18, 2021

*: Data collection from FCOS machines #31

Are you sure you want to change the base?

*: Data collection from FCOS machines #31

Conversation

zonggen commented Sep 18, 2019 • edited Loading

zonggen commented Sep 18, 2019

zonggen commented Sep 18, 2019

zonggen commented Sep 19, 2019 • edited Loading

jlebon left a comment

Choose a reason for hiding this comment

zonggen commented Oct 15, 2019 • edited Loading

zonggen commented Oct 21, 2019

zonggen commented Oct 22, 2019

zonggen commented Oct 24, 2019

zonggen commented Oct 28, 2019

ashcrow commented Mar 18, 2021

bgilbert commented Mar 18, 2021

zonggen commented Sep 18, 2019 •

edited

Loading

zonggen commented Sep 19, 2019 •

edited

Loading

zonggen commented Oct 15, 2019 •

edited

Loading