Skip to content

Latest commit

 

History

History
432 lines (282 loc) · 18.1 KB

Changelog.md

File metadata and controls

432 lines (282 loc) · 18.1 KB
Raw

Deployment Changelog

Changes in this log refer only to changes that make it to the 'main' branch and are nested under deploy/cttso-ica-to-pieriandx-cdk.

2024-05-27

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Hotfix

  • Debug authentication issues (#229)

2024-05-23

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Hotfixes

  • Improve access token handling (#226)

2024-05-08

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Hotfixes

  • Excel interprets a few portal run ids as numbers i.e 2021121980805e78

2024-05-07

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Enhancements

  • Add ctTSO 1.2 portal run id to the LIMS sheet (#216)

  • Move to pieriandx infrastructure secret (#221)

Configuration Changes

  • Update project name mapping (#217)

Documentation

2024-02-02

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Dependencies

  • Ensure gspread is less than version 6.0.0 (#208)

2023-11-27

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Hotfixes

  • lims maker lambda fails on type hinting for multiple types (#201)

Enhancements

  • Use global var for NTC Subject ID (#202)

2023-11-22

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Hotfixes

  • Update disease name to disease label (#188)
  • Coerce is_identified to boolean (#187)
  • Use comma after needs_redcap column (#186)
  • Use non-latest docker tag. (#191)
  • Fix deidentified samples where wrong keys used for study id and participant id (#194)
  • Fix deidentified samples where NAT instead of NaT null value was used (#198)

Workarounds

  • Don't submit NTC samples (#195)

2023-11-08

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Hotfixes

  • Update date columns check (#174)

2023-11-07

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Enhancements

  • Add two pieriandx columns for disease code and names (#167)

Bugfixes

  • Refactored validation lambda so that checks are not done before columns are renamed (#168)

  • Don't delete pending cases from cttso lims (#169)

  • Added snomed term to submission df (#170)

2023-10-18

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Bugfixes

  • Dont resubmit deleted samples - also remove deleted samples from the lims sheet (#151)
  • Use loc over iloc for pandas selection (#154)
  • Replace mrn with external_subject_id for list of required fields (#159)

2023-10-18

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Enhancements

  • Move to project owner / project name mapping logic (#141)

  • Add deleted sheet (#140)

    • All cases assigned to user ToBe Deleted, are moved to a separate sheet

2023-08-13

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

  • Updated submission columns

Hot fixes

  • Updated submission column #124 (fixes #123)
  • Update is_validation logic #127 (fixes #126)

2023-08-09

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

  • Improved diagram for automation pathway

Bug fixes

  • Make sure RedCap Sample Type is used if RedCap Entry is not null
  • Ensure Lambda Pathways are as expected on the diagram
    • GLIMS ProjectName is set to Validation or Control
    • OR
      • Sample NOT in RedCap
      • AND
      • GLIMS Workflow is set to 'Research'

2023-07-31

Author: Alexis Lucattini Email: Alexis.Lucattini@ummcr.org

HotFix

  • Fix submission time issue when merge adds suffixes to columns (#118)

Bugfix

  • Fix cancelled cases when workflow is still NA (so needs an update) (#117)

2023-07-17

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Hot fixes

  • Fixed issue where collection of pieriandx submission time resulted in error when pieriandx case id was null (#112)

Bug fixes

  • Fixed issue where changelog for main changelog was being skipped (#112)

2023-07-10

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Bug fixes

  • Gh Actions 'Check Changelog' was not using the correct attributes (#96)
  • Check comparing portal workflow run end date and pieriandx submission date should be done in EST time (#104)
    • Resolves #101
    • Resolves #100
    • This issue was likely the cause of many 'duplicate' submissions
  • Update the pieriandx_submission_time column using processing_df (#103)
  • Use subject id, library id and portal run id to submission time from lims df (#106)

Enhancements

  • Use pieriandx sample type column in LIMS df (#105)

2023-06-28

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Enhancements

  • Batch Stack Upgrade (#79)

    • Resolves (#76)

  • Upgrade CDK to version 2.85.0 (#79)

    • Resolves (#78)

Fixes

  • Lambda utils miscell python file was importing the wrong get_logger function (#82)

    • Resolves (#81)

  • Use Job ARN over Job Name for batch submission from lambda (#91)

  • Create disable rule policy document as a separate entity (#93)

  • Create SSM Get parameter policy as separate entity (#95)

Various fixes after migration to new aws-batch-alpha

2023-06-23

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Fixes

  • Fix Type Hinting for EventClient Type (#71)
  • Don't use RequestResponse event type when manually deploying lambdas (#64)
  • Get Incomplete Job Df from Gsuite doesn't collect pending jobs at times (#66

Enhancements

  • Added GH Actions check to ensure this changelog file has been updated before deployment (#73)

  • Added EventBridge kill switch if processing_df contains items included in the update_df (shouldn't happen) (#67)

  • Check if new password can successfully generate pieriandx session token before successfully exiting script (#57)

Dependabot Updates

  • Migrated from AWS 2.39.1 to 2.80.0 (#74)

2023-04-14

  • Prevent NTCs from being uploaded to PierianDx (#52)

  • Add new PierianDx Submission Time Column to LIMS Sheet (#47)

  • Handle submission edge case of SBJ01666 (#50)

2023-02-15

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Fixes

  • Bugfix for redcap_is_complete column in ctTSO LIMS lambda (#23)
  • Fix issue if cttso lims is empty (from reset)
  • Panel type needed axis=columns keyword argument
  • submission function needs to consider panel type as an option (#32)
  • Use Workflow column name over ProjectName in GLIMS to determine column type (#24)
    • Fixed by #29
    • Updated flowchart to reflect this change (#30)
  • Fix Research Sample not completing in validation pipeline edge case (#38)
  • Fix unhelpful merged row logging error (#39)
  • Removed debug line in portal helpers code (#40)
  • Fixed issue where ctTSO lims was not always collecting the latest job or report (#41)

Enhancements

  • Allow for 20 lambda submissions per cycle (#28)
  • Update lambda layers requirements file with updated boto3 and botocore to allow for new sso config syntax (#25) and (#36)

2023-01-23

Author: Alexis Lucattini Email: Alexis.Lucattini@ummcr.org

Updates

  • Updated panel from VALIDATION, PATIENTCARE names to MAIN, SUBPANEL respectively (#19)
    • This change better reflects the flexibility between panel and sample types.
    • Some clinical samples or research samples may need to go through the main panel
    • We may also wish to test the subpanel with validation samples
  • Added glims_is_research column to cttso lims (if glims matches 'research' in the ProjectName column)
    • If glims_is_research is True then a sapmle is processed through the MAIN panel (even if it's set as a clinical sample in RedCap)
  • Added panel type column to cttso lims (scraped from pieriandx)
  • Added panel_type option to clinical and research payload parameters
    • Default payload for clinical is "SUBPANEL"
    • Default payload for validation is "MAIN"\

2022-11-16

Author: Alexis Lucattini Email: Alexis.Lucattini@ummcr.org

Fixes

  • Reduce number of items returned by portal from 1000 to 100 to prevent internal server errors.
  • Version control gspread to prevent pip install failure for lambdas
  • Reduce number of submissions to pieriandx to reduce timeouts

2022-09-22

Author: Alexis Lucattini Email: Alexis.Lucattini@ummcr.org

Wake up Lambdas

  • Lambdas are woken up before payloads are submitted, documentation added to deployment ReadMe

Run jobs sequentially

  • Set max availble vCPUs to 3 and specify 2 cpus per batch run instance

2022-08-30

Author: Alexis Lucattini Email: Alexis.Lucattini@umccr.org

Summary

  • Created a suite of lambdas to improve automatic processing of pieriandx data from ICA

    • Launch validation workflow - takes data from portal and defaults and creates case in PierianDx
    • Create a Google SpreadSheet using APIs from redcap, portal, pieriandx and GLIMS inputs.
      • This can also find available launch payloads and launches them accordingly

  • Created a token lambda that creates a new pieriandx authentication token every five minutes

  • Restructured the pipeline stack to include new lambda and removed some stages outside of the wave due to dependency chains

  • Set tags to have key 'Stack' rather than prefix-Stack

  • Updated user to create cases in PierianDx from alexisl@unimelb.edu to services@umccr.org

  • Fixed issue for users wanting to launch redcap payloads manually

  • Updated to cdk 2.39.1