Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ADI retrieval from a source on S3 #1851

Closed
iamleeg opened this issue May 12, 2021 · 1 comment
Closed

Allow ADI retrieval from a source on S3 #1851

iamleeg opened this issue May 12, 2021 · 1 comment
Assignees
Labels
P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix
Milestone

Comments

@iamleeg
Copy link
Contributor

iamleeg commented May 12, 2021

Is your feature request related to a problem? Please describe.
We have secret squirrel data sources that we're not allowed to share publicly like GISAID. But to ingest from them, they currently have to be visible at a publicly-available HTTP[S] URI, which means we have to share them publicly.

Describe the solution you'd like
We should be able to point to an S3 resource as a source and have the retrieval function fetch data from there to trigger the parser.

Describe alternatives you've considered
Anything that is access-controlled where we can store the authentication secrets secretly(!) would work, and S3 is probably the easiest because we already use it in the ADI workflow so the access issue has already been solved.

@iamleeg iamleeg added the P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix label May 12, 2021
@iamleeg iamleeg added this to the Strongcat milestone May 12, 2021
@iamleeg iamleeg self-assigned this May 18, 2021
@iamleeg iamleeg assigned iamleeg and unassigned iamleeg Jun 7, 2021
iamleeg added a commit that referenced this issue Jun 8, 2021
@abhidg
Copy link
Contributor

abhidg commented Jun 12, 2021

Works great! Tested on dev, but should work on prod as well as ingestion code is always pulled from main.

@abhidg abhidg closed this as completed Jun 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix
Projects
None yet
Development

No branches or pull requests

2 participants