Ticket/2450/supplemental/columns

[danielsf]

As part of the 2022 VBN release, we are adding some hand annotations not currently represented in the LIMS database to the ecephys_sessions table. Rather than update the schema of the LIMS database (a change that would have implications for all previously-collected ecephys data), we are adding functionality to the VBN metadata_writer that allows us to add columns to the ecephys_sessions table by hand. This PR encompasses that functionality. The new supplemental_columns entry in the metadata_writer schema should look something like this

  "supplemental_columns": [
    {
      "abnormal_activity": false,
      "ecephys_session_id": 1051155866
    },
    {
      "abnormal_activity": false,
      "ecephys_session_id": 1044385384
    },
    {
      "abnormal_activity": false,
      "ecephys_session_id": 1044594870
    },
    {
      "abnormal_activity": false,
      "abnormal_histology": [
        "Hippocampus"
      ],
      "ecephys_session_id": 1056495334
    }]

[aamster]

Can't you just use pd.merge here instead of patch_df_from_other?

[danielsf]

Even if we did just use pd.merge, I'd want to wrap it in a function that we could test to make sure that the columns we are adding get added the way we expect. patch_df_from_other is already tested. I'd rather keep this as it is.

[aamster]

I think this argument would be better named as supplemental_data. supplemental_columns makes it seem like it is a list of column names.

[danielsf]

done

[aamster]

Should this rather be an input file? Passing a long list of dicts through the command line would be cumbersome.

[danielsf]

I don't want to proliferate the number of input files we have to keep track of. These input.jsons are large, but they have the virtue of carrying everything we need in one package.

I also don't see users specifying this field (or, or that matter, probes_to_skip) on the command line.

I'd rather leave this as it is.