Merge pull request #68 from tuva-health/refactor-connector

Refactor connector: Add adjustment logic and better enrollment dates

README.md

@@ -2,11 +2,11 @@
 
 # Medicare CCLF Connector
 
-## 🔗  Docs
+## 🔗 Docs
 Check out our [docs](https://thetuvaproject.com/) to learn about the project and how you can use it.
 <br/><br/>
 
-## 🧰  What does this repo do?
+## 🧰 What does this repo do?
 
 The Medicare CCLF Connector is a dbt project that maps raw Medicare CCLF claims data to the Tuva Input Layer, which is the first step in running the Tuva Project.  This connector expects your CCLF data to be organized into the tables outlined in this [CMS data dictionary](https://www.cms.gov/files/document/cclf-information-packet.pdf), which is the most recent format CMS uses to distribute CCLF files.
 <br/><br/>  
@@ -18,7 +18,7 @@ The Medicare CCLF Connector is a dbt project that maps raw Medicare CCLF claims
 - Snowflake
 <br/><br/>  
 
-## ✅  Quickstart Guide
+## ✅ Quickstart Guide
 
 ### Step 1: Clone or Fork this Repository
 Unlike [the Tuva Project](https://github.com/tuva-health/the_tuva_project), this repo is a dbt project, not a dbt package.  Clone or fork this repository to your local machine.
@@ -29,21 +29,55 @@ Next you need to import the Tuva Project dbt package into the Medicare CCLF Conn
 <br/><br/> 
 
 ### Step 3: Data Preparation
-The CCLF file specification does not have a field that can be mapped directly to `enrollment_start_date`; therefore, we have added a field called `bene_member_month`. We recommend parsing the monthly enrollment file date from the Beneficiary Demographics filename (e.g., P.A****.ACO.ZC8Y**.Dyymmdd.Thhmmsst) and mapping this date to `bene_member_month`. The connector will handle the rest of the mapping.
+
+#### Source data:
+The source table names the connector is expecting can be found in the 
+`_sources.yml` config file. You can rename your source tables if needed or add an 
+alias to the config.  
+
+#### File Dates:
+The field `file_date` is used throughout this connector to deduplicate data 
+received across regular and run-out CCLFs. We recommend parsing this date from 
+the filename (e.g., P.A****.ACO.ZC1Y**.Dyymmdd.Thhmmsst) and formatting it as 
+"YYYY-MM-DD".
+
+#### Enrollment Dates:
+The CCLF specification does not have a field that can be mapped directly 
+to `enrollment_start_date` and `enrollment_end_date`, and the Part A and Part B 
+entitlement dates (BENE_PART_A_ENRLMT_BGN_DT, BENE_PART_B_ENRLMT_BGN_DT) are 
+often incorrect or not useful for claims analytics.
+
+We have included an additional source called `Enrollment` that can be
+populated with enrollment dates relevant to your data. These enrollment
+dates may come from an attribution file, beneficiary alignment report (BAR), or
+any source you may have. You just need to create a source table with the 
+following columns:
+
+  1. `current_bene_mbi_id`
+  2. `enrollment_start_date`
+  3. `enrollment_end_date`
+  4. `bene_member_month`
+     * The connector includes logic to handle enrollment spans or member months.
+     * If enrollment spans are available, leave this field null.
+     * If enrollment spans are not available, populate this field with member 
+       month dates in the format "YYYY-MM-DD" and set the variable 
+       `member_months_enrollment` to true in the `dbt_project.yml` file.
 <br/><br/> 
 
 ### Step 4: Configure Input Database and Schema
 Next you need to tell dbt where your Medicare CCLF source data is located.  Do this using the variables `input_database` and `input_schema` in the `dbt_project.yml` file.  You also need to configure your `profile` in the `dbt_project.yml`.
 <br/><br/> 
 
 ### Step 5: Run
-Now you're ready to run the connector and the Tuva Project.  For example, using dbt CLI you would `cd` to the project root folder in the command line and execute `dbt build`.  Next you're now ready to do claims data analytics!
+Finally, run the connector and the Tuva Project. For example, using dbt CLI you would `cd` to the project root folder in the command line and execute `dbt build`.  
+
+Now you're ready to do claims data analytics!
 <br/><br/>
 
-## 🙋🏻‍♀️ How do I contribute?
+## 🙋🏻‍♀️ How do I contribute?
 Have an opinion on the mappings? Notice any bugs when installing and running the project?
 If so, we highly encourage and welcome feedback!  While we work on a formal process in Github, we can be easily reached on our Slack community.
 <br/><br/>
 
-## 🤝 Join our community!
+## 🤝 Join our community!
 Join our growing community of healthcare data practitioners on [Slack](https://join.slack.com/t/thetuvaproject/shared_invite/zt-16iz61187-G522Mc2WGA2mHF57e0il0Q)!

dbt_project.yml

@@ -7,11 +7,14 @@ profile: default
 vars:
 
   ## Set the database and schema where raw CCLF data is stored
-  input_database: dev_ci_testing
-  input_schema: cclf
+  input_database: tuva
+  input_schema: raw_data
 
   claims_enabled: true
 
+  ## Uncomment this variable if enrollment source is member months
+#  member_months_enrollment: true
+
 model-paths: ["models"]
 analysis-paths: ["analyses"]
 test-paths: ["tests"]
@@ -24,12 +27,11 @@ clean-targets:
   - "target"
   - "dbt_packages"
 
-# models:
-#   medicare_cclf_connector:
-#     +database: default
-#   the_tuva_project:
-#     +database: default
-
-# seeds:
-#   +database: default
-
+models:
+   medicare_cclf_connector:
+     final:
+       +materialized: table
+     intermediate:
+       +materialized: table
+     staging:
+       +materialized: view