We need a more robust interface for ways of (a) processing numerical and categorical values and (b) normalizing output data in light of those modes.

[mmcdermott]

This includes both some discussion around how we should structure these stages and some key implementation details. More details forthcoming.

• Once quantiles are added, we need to be able to support binning numerical values as well #99
• Normalize transform should be separated and configurable for different normalization modes, between normalizing numerical values and codes, etc. #65
• Adds an 'extract_values' transform to extract values and retype them from input MEDS data. #121 (possibly)

[mmcdermott]

As of now, I'm thinking we need the following capabilities:

1. Numeric values are extracted from text. (e.g., blood pressure recorded as "120/80"). This is part of Adds an 'extract_values' transform to extract values and retype them from input MEDS data. #121
2. Numeric values are converted to part of the code in raw form (e.g., Glasgow Coma Score is recorded numerically, but we want to embed it categorically). This is part of Adds an 'extract_values' transform to extract values and retype them from input MEDS data. #121
3. Numeric values are normalized to have zero mean and unit variance. This is currently supported, and revisions to that API are handled in Normalize transform should be separated and configurable for different normalization modes, between normalizing numerical values and codes, etc. #65 -- I suspect the right way to do this is to have one stage that computes means and variances on just the metadata.parquet file alone, then another stage that uses them to do the normalization. Right now these both happen in the same stage. Other normalization strategies (e.g., min-max) should likewise be added and are covered in Normalize transform should be separated and configurable for different normalization modes, between normalizing numerical values and codes, etc. #65
4. Numeric values are binned into buckets. Then, these are
   • Combined with code (this is used). The codes.parquet metadata file must be updated when this happens or in an immediate subsequent stage.
   • Added as a new event with just a value-specific code (this is used in ETHOS). The codes.parquet metadata file must be updated when this happens or in an immediate subsequent stage.
   • Kept as a numerical value?? (I don't know if this is used)

[mmcdermott]

@Oufattole @prenc does this sound right to you?

[Oufattole]

Thanks Matthew great summary!

Looks good to me, my only question is how will (4) maintain that time derived features should precede all other features, or lexicographic ordering computed in a previous vocab_indices stage? Would a user be expected to run a vocab_indices stage again after?

[mmcdermott]

The lexicographic ordering is just for the code vocab indices. If you want to ensure that time derived codes come first you should use the reorder stage.

…

On Sun, Aug 25, 2024, 2:02 PM Oufattole ***@***.***> wrote: Thanks Matthew great summary! Looks good to me, my only question is how will (4) maintain that time derived features should precede all other features, or lexicographic ordering computed in a previous vocab_indices stage? Would a user be expected to run a vocab_indices stage again after? — Reply to this email directly, view it on GitHub <#177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADS5X6IXSQPRUARL7DCSB3ZTIL2BAVCNFSM6AAAAABNCXIGJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBYHE2DEOJUGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mmcdermott]

fyi @Oufattole and @prenc #57 has been closed and now in MEDS v0.0.6 on pypi you can run an extract_values stage which will enable you to convert numeric values to categorical values and vice versa as desired. See an example (I know, documenation is forthcoming) here: https://github.com/mmcdermott/MEDS_transforms/blob/main/tests/test_extract_values.py