Server: Fedora CoreOS Pinger Backend Design

[zonggen]

After sending the data, we need an endpoint to collect the data sent from client side.

Current plan is to run a simple HTTP server inside a container and aggregate the data sent from clients according to different time intervals (daily vs. monthly).

After receiving the data, the server would store the information depending on the information types (e.g. platform_id, os_version etc.) and count the number of instances that has this information. A report script that summarizes this information would be helpful too.
An example would be:

$ cat /path/to/collected_info.json
{
    "platform_id": {
                          "qemu": 4312,
                          "openstack": 1234,
                           "gcp": 5432
                   },
    "current_os_version": {
                          "30.20190905.dev.2 (CoreOS preview)": 1234,
                           "30.20190905.dev.3 (CoreOS preview)": 1234
                          },
...
}

[lucab]

I'm leaving here a single comment, but this covers all of #30, #32 and #33.

I'd rather suggest that we start the other way around (i.e. from the intended usecases for this data), until we finally reach the format on-wire, the backend implementation and the database.

That is, we should ask ourselves and answer in order:

1. which kind of queries do we plan to perform on the collected data? How do we roll those across sliding time-windows?
2. in order to satisfy the above, which granularity/retention should we maintain when persisting the collected data? Do we need to periodically consolidate and scrub data?
3. in order to satisfy the above, how should we pick/organize the data-schema and database?
4. in order to satisfy the above, how does the on-wire format looks likes?
5. in order to satisfy the above, how do we encode/normalize the structured data we collect?

[lucab]

@zonggen as you updated the ticket in the meanwhile, let me map my comment to your example. Are we interested in answering things like "how many GCP nodes actively reported in the last two weeks" or "how many openstack nodes reported yesterday with a non-latest OS version"?
As we are designing this system primarily for our own consumption, we should make sure we are recording all the data with the granularity we need to answer whatever questions we may be interested in.

[zonggen]

To summarize my understanding, our main purpose of this project is to know (from coreos/fedora-coreos-tracker#86 (comment))

what platforms or container runtimes to prioritize; what hardware, system services, or system or network configurations are commonly used; which corner cases need to be especially well tested during upgrades; and which third-party services provide important compatibility constraints

Since different information is gathered under the flag minimal and full, the queries we are running is different. For minimal, we are only counting the numbers of instances, for example, how many instances are running on gcp, and how many instances are running in version 30.20190905.dev.2 (CoreOS preview) etc. For full, we are collecting more detailed information, such as a summary of network configuration, a summary of hardware etc and such information might require creation of user object on database (without creation of unique uuid).

For database schema, two collections can be created: minimal and full. For minimal, we could store the key/value pairs that count the amount of instances. For full, we are storing the data under the user object, again no unique uuid is recorded and thus no concerns of long term tracking. No-SQL database could be used, like MongoDB(https://docs.rs/mongodb/0.4.0/mongodb/).

We could use json format when sending the data (or any other format with key/value storage) and make a HTTP POST request to server.

[zonggen]

As we are designing this system primarily for our own consumption, we should make sure we are recording all the data with the granularity we need to answer whatever questions we may be interested in.

I think the scale of flexibility of answering such questions depends on how organized/well-designed our database is. For example, "how many GCP nodes actively reported in the last two weeks" can be retrieved from our database by "sum(platform_id='gcp' where d1 <= date <= d2) " and "how many openstack nodes reported yesterday with a non-latest OS version" by "sum(platform_id='openstack' where date=d and os_release != latest_ver)".

[zonggen]

One question for the backend would be, how are we going to make sense of the data under the flag full since there is no counting for the specific configuration (we are collecting the verbal summary of specifications)? When we are trying to query the database, we will need to look for a specific specification among all users under certain interval and count the instances, which is okay but could be improved..