Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/save metadata to firebase #206

Merged

Conversation

rugeli
Copy link
Collaborator

@rugeli rugeli commented Oct 23, 2023

Problem

What is the problem this work solves, including
closes #194

Solution

What I/we did to solve this problem

  • wrote a function to save metadata of aws results files to firebase. The stored metadata includes fields such as "username", "timestamp", "aws_url", with the recipe's unique name being as the key
  • handlers are initiated using the DATABASE_IDS enum, ensuring that the DBRecipeHandler remains agnostic
  • rearranged firebase handler and aws handler to
    • ensure they are initialized only once
    • improve readability

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

Steps to Verify:

  1. pack either local or remote recipes to see if the result files has been uploaded to S3 and saved metadata to firebase

Screenshots (optional):

Screenshot 2023-10-23 at 1 16 59 PM

@github-actions
Copy link

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name Encapsulating radius Average number packed
ext_A 25 236.0

Packing image

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key Pairwise distance distribution
ext_A Distance distribution ext_A

@rugeli
Copy link
Collaborator Author

rugeli commented Oct 23, 2023

The previous PR#201 was closed due to some merging conflicts and rebasing issues. This new PR has the necessary changes for saving metadata and offers a cleaner commit history for review. It will be merged into #179, which is being treated as a staging branch now.

@rugeli rugeli requested review from meganrm and mogres October 23, 2023 20:35
Copy link
Collaborator

@mogres mogres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I was able to run a local recipe and view the uploaded metadata on firebase.

Copy link
Member

@meganrm meganrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@rugeli rugeli merged commit 6f01cfd into feature/run-recipes-from-firebase Nov 6, 2023
2 checks passed
@rugeli rugeli deleted the feature/save-metadata-to-firebase branch November 6, 2023 22:52
rugeli added a commit that referenced this pull request Dec 18, 2023
* [wip] prep recipe data for packing

* get creds from local file

* save firebase creds to a .creds file

* remove cred arg

* check for already handled values in remote recipes

* can pack one sphere

* * adding username to .creds
* formatting

* move `write_username_to_creds`

* download recipe testing

* edit comment

* code refactor

* lint

* format tests

* add prep_db_doc

* changed class name in DBRecipeHandler

* fix lint and test errors

* initialize firebase handler only once

* refactor message

* add remote db options in `pack`

* remove a print statement

* rename and reorg DB handler

* fix tests

* move database_ids enum to interface_objects

* remove db_handler in pack and recipe_loader

* send db_handler in to autopack

* rename functions

* integrate DATABASE_NAMES into interface_objects

* lint

* Feature/run inherited objects (#198)

* turn off resolving inheritance while uploading

* able to upload recipes having "inherit" key

* get download and pack to work, refactors needed

* refactors

* formatting

* testing and refactor

* Feature/save metadata to firebase (#206)

* refactor AWS and firebase handler

* databases initiation handling

* refactor

* Update .gitignore

Co-authored-by: Saurabh Mogre <saurabh.mogre@alleninstitute.org>

* add file existence check

* refactor is_nested_list method

* revert write_json_file

* formatting

---------

Co-authored-by: meganrm <meganrm@gmail.com>
Co-authored-by: Saurabh Mogre <saurabh.mogre@alleninstitute.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants