Match MEDS label schema as per #72

[justin13601]

• Renamed index_timestamp as part of Match the MEDS label schema for MEDS dataset #72

Summary by CodeRabbit

• New Features

  • Added validation for MEDS label DataFrames to ensure compliance with specified schemas before processing.
  • Introduced a new dependency on the meds library, enhancing project capabilities.
  • Updated expand_shards function to utilize modern file handling techniques and provide clearer output formats.

• Bug Fixes

  • Improved handling of DataFrame columns by ensuring the "index_timestamp" column is only renamed to "prediction_time" if it exists, reducing potential errors.
  • Modified time column handling in data processing, ensuring consistent naming and conversion.

[coderabbitai]

Walkthrough

The recent update significantly enhances data handling capabilities within the src/aces/__main__.py, src/aces/expand_shards.py, and src/aces/predicates.py files. Key improvements include robust validation of MEDS label data, modernized file discovery methods, and refined handling of time data. These updates lead to more reliable, maintainable code while adhering to specified schemas and improving user experience.

Changes

Files	Change Summary
src/aces/main.py, src/aces/predicates.py	Enhanced MEDS label validation with get_and_validate_label_schema; renamed columns for clarity and consistency.
src/aces/expand_shards.py	Updated expand_shards to use Path.glob for improved directory handling; changed output from string to sorted list of paths.
pyproject.toml	Added "meds == 0.3" to dependencies to enhance project capabilities.

Poem

🐇 In the code where logic flows,
A rabbit hops where data grows.
With checks to guard, no errors to fear,
Renamed it well, so all is clear!
Hooray for changes, let’s all cheer,
For the magic of coding brings us near! ✨

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[codecov]

Codecov Report

Attention: Patch coverage is 28.57143% with 30 lines in your changes missing coverage. Please review.

Files	Patch %	Lines
src/aces/__main__.py	23.07%	30 Missing ⚠️

Files	Coverage Δ
src/aces/expand_shards.py	90.47% <100.00%> (-0.44%)	⬇️
src/aces/predicates.py	83.07% <100.00%> (ø)
src/aces/__main__.py	56.33% <23.07%> (-37.95%)	⬇️

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between ff5efab and b52a3ae.

Files selected for processing (1)

• src/aces/main.py (1 hunks)

Additional context used

GitHub Check: codecov/patch

src/aces/__main__.py

[warning] 50-51: src/aces/main.py#L50-L51
Added lines #L50 - L51 were not covered by tests

Additional comments not posted (1)

src/aces/__main__.py (1)

50-51: LGTM! Add tests for the new conditional logic.

The added conditional check ensures that the renaming operation is only performed if the "index_timestamp" column exists, which enhances robustness. However, the new logic is not covered by tests.

Would you like me to assist in generating the unit tests or open a GitHub issue to track this task?

Tools

GitHub Check: codecov/patch

[warning] 50-51: src/aces/main.py#L50-L51
Added lines #L50 - L51 were not covered by tests

[mmcdermott]

@justin13601 We'll probably need to extend this to encompass other changes to match the label schema introduced here: Medical-Event-Data-Standard/meds#32

That PR is now merged, so we can reliably match MEDS v0.3 at this point.

It would be worth explicitly adding a pyproject.toml dependency on meds at version 0.3 (once it is pushed) and using that to either fully or partially (partially if we include an inclusive or exclusive flag on the prediction_time) validate the output schema as well.

[mmcdermott]

There is more that we need to change for this; I think the right strategy is to emulate the code I linked in the related issue to this PR. That will handle both column names and pyarrow types.

[coderabbitai]

Actionable comments posted: 3

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between b52a3ae and 0cde8f6.

Files selected for processing (3)

• pyproject.toml (1 hunks)
• src/aces/main.py (3 hunks)
• src/aces/expand_shards.py (3 hunks)

Additional context used

GitHub Check: codecov/patch

src/aces/__main__.py

[warning] 85-91: src/aces/main.py#L85-L91
Added lines #L85 - L91 were not covered by tests

---

[warning] 93-94: src/aces/main.py#L93-L94
Added lines #L93 - L94 were not covered by tests

---

[warning] 96-100: src/aces/main.py#L96-L100
Added lines #L96 - L100 were not covered by tests

---

[warning] 102-102: src/aces/main.py#L102
Added line #L102 was not covered by tests

---

[warning] 138-142: src/aces/main.py#L138-L142
Added lines #L138 - L142 were not covered by tests

Additional comments not posted (2)

pyproject.toml (1)

25-25: Dependency addition looks good.

The addition of the meds == 0.3 dependency aligns with the comments summary and supports the validation of the MEDS label schema.

src/aces/expand_shards.py (1)

6-6: Good use of Path for file handling.

The use of Path.glob enhances the robustness and readability of the code for file discovery.

[coderabbitai]

Actionable comments posted: 3

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 0cde8f6 and e965539.

Files selected for processing (3)

• src/aces/main.py (3 hunks)
• src/aces/expand_shards.py (3 hunks)
• src/aces/predicates.py (2 hunks)

Files skipped from review as they are similar to previous changes (1)

• src/aces/expand_shards.py

Additional context used

GitHub Check: codecov/patch

src/aces/__main__.py

[warning] 85-87: src/aces/main.py#L85-L87
Added lines #L85 - L87 were not covered by tests

---

[warning] 93-98: src/aces/main.py#L93-L98
Added lines #L93 - L98 were not covered by tests

---

[warning] 100-101: src/aces/main.py#L100-L101
Added lines #L100 - L101 were not covered by tests

---

[warning] 103-107: src/aces/main.py#L103-L107
Added lines #L103 - L107 were not covered by tests

---

[warning] 109-109: src/aces/main.py#L109
Added line #L109 was not covered by tests

---

[warning] 112-114: src/aces/main.py#L112-L114
Added lines #L112 - L114 were not covered by tests

---

[warning] 119-119: src/aces/main.py#L119
Added line #L119 was not covered by tests

---

[warning] 121-121: src/aces/main.py#L121
Added line #L121 was not covered by tests

---

[warning] 125-125: src/aces/main.py#L125
Added line #L125 was not covered by tests

---

[warning] 161-167: src/aces/main.py#L161-L167
Added lines #L161 - L167 were not covered by tests

Additional comments not posted (4)

src/aces/__main__.py (3)

22-32: Constants for MEDS label types are well-defined.

The constants MEDS_LABEL_MANDATORY_TYPES and MEDS_LABEL_OPTIONAL_TYPES correctly specify the expected data types for MEDS label DataFrames.

---

35-125: Function logic for schema validation is robust.

The get_and_validate_label_schema function effectively ensures that the DataFrame conforms to the expected MEDS schema.

Tools

GitHub Check: codecov/patch

[warning] 85-87: src/aces/main.py#L85-L87
Added lines #L85 - L87 were not covered by tests

---

[warning] 93-98: src/aces/main.py#L93-L98
Added lines #L93 - L98 were not covered by tests

---

[warning] 100-101: src/aces/main.py#L100-L101
Added lines #L100 - L101 were not covered by tests

---

[warning] 103-107: src/aces/main.py#L103-L107
Added lines #L103 - L107 were not covered by tests

---

[warning] 109-109: src/aces/main.py#L109
Added line #L109 was not covered by tests

---

[warning] 112-114: src/aces/main.py#L112-L114
Added lines #L112 - L114 were not covered by tests

---

[warning] 119-119: src/aces/main.py#L119
Added line #L119 was not covered by tests

---

[warning] 121-121: src/aces/main.py#L121
Added line #L121 was not covered by tests

---

[warning] 125-125: src/aces/main.py#L125
Added line #L125 was not covered by tests

---

160-167: MEDS data processing logic in main is correct.

The function correctly renames columns and validates the schema for MEDS data.

Tools

GitHub Check: codecov/patch

[warning] 161-167: src/aces/main.py#L161-L167
Added lines #L161 - L167 were not covered by tests

src/aces/predicates.py (1)

Line range hint 294-319: Column renaming in generate_plain_predicates_from_meds is consistent with MEDS schema.

The renaming of columns from "time" to "timestamp" and "patient_id" to "subject_id" aligns with the schema changes.