Design / make recommendations for automating manual steps in current ETL process

[pieterlukasse]

The goal of this ticket is to explore and recommend potential solutions for the parts of the ETL process that are done manually now. These parts are marked in the diagram below (also added as an image below this ticket):

• link to high res version Google Drawing: https://docs.google.com/drawings/d/1HaPPdGOr7VaUbRe6pEzQmAiT53Cq_jXsIl6-eajo1XE/edit

Important details to consider:

• some parts might have been automated as part of https://github.com/d3b-center/cbioportal-loading-automation, so review that first
• a part of the process is already automated as a github action (bottom right corner of the diagram). It is worth exploring whether the other steps can also move into that, or use the same/similar infrastructure.
• work with Alex Lubneuski to validate / assess feasibility of proposed solutions
• keep Moving/inconsistent data locations #48 in mind
• consider/ask about option of using Cavatica for all genomics files (instead of the various S3 buckets)

[migbro]

Using CAVATICA for all genomics files has become sort of a reality currently as @zhangb1 and his team have been more on top of keeping those links up-to-date