Flatten _validation.toml

[chStaiger]

Reading in the validation.toml leads to a very nested dictionary and list of dictionaries. We need to discuss how to flatten that a bit.
@StefanoRapisarda can you please have a look in the code how I am using the toml and give some suggestions how to flatten the data structure a bit?

[chStaiger]

Here is how I read in the toml file:

autumn-fair/src/check_data.py

Line 95 in 200f234

val_path = Path("data/metadata_test/_validation_schema.toml") # param for client

And this is the code to get the information into a workable python data structure:

autumn-fair/src/check_data.py

Line 12 in 200f234

def flatten_list_of_dicts(key: str, val: str, data: list) -> dict:

[StefanoRapisarda]

I made a new version of metadata for validation ("_validation_schema_v2.toml") and renamed the old one (v1).

This is an example of the v2 file structure:

[hosts]
host_id.type = "string"
host_id.format = "AA0_00000"
host_groupNumber.type = "integer"
host_sex.type = "string"
host_sex.format = "A"
host_sex.values = ["M","F"]
host_age.type = "integer"
host_death.type = "integer"
host_species.type = "string"
host_breed.type = "string"

The levels are <file_name> --> <column_name> --> <column_attribute>. So to access the type of the host_id column in the events file you should use metadata["events"]["host_id"]["type"]

Not all the columns have all the possible attributes, so I added the complete list of possible attributes at the very beginning of the file (metadata_keys). Compared to the previous version, this structure removes a layer, with the caveat that now you need to check for keywords listed in metadata_keys