Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/upload local recipes to s3 #189

Merged
merged 6 commits into from
Sep 22, 2023
Merged

Conversation

rugeli
Copy link
Collaborator

@rugeli rugeli commented Aug 29, 2023

Problem

What is the problem this work solves, including
Partially closing #140

Solution

What I/we did to solve this problem

  • Updated simularium_helper.py to:
    - store local results to S3 path: cellpack-results/simularium
    - enable automatically opened packing results on Simularium
  • Updated README

Note: a separate PR will handle the upload of metadata to firebase

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

Steps to Verify:

  1. Pack a local recipe: pack -r cellpack/tests/recipes/v2/test_spheres.json -c examples/packing-configs/sparse-packing.json
  2. Check the uploaded object in S3 bucket, should look like this:
Screenshot 2023-08-29 at 3 00 31 PM

Keyfiles (delete if not relevant):

  1. simularium_helper.py
  2. AWSHandler.py

@codecov-commenter
Copy link

codecov-commenter commented Aug 29, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (2c9db6d) 98.52% compared to head (2b22d68) 98.52%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #189   +/-   ##
=======================================
  Coverage   98.52%   98.52%           
=======================================
  Files          16       16           
  Lines         476      476           
=======================================
  Hits          469      469           
  Misses          7        7           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions
Copy link

github-actions bot commented Aug 29, 2023

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name Encapsulating radius Average number packed
ext_A 25 236.0

Packing image

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key Pairwise distance distribution
ext_A Distance distribution ext_A

@rugeli
Copy link
Collaborator Author

rugeli commented Aug 29, 2023

This PR focuses on uploading local recipes to S3 and is ready for prioritized merging after review. The database handler import issue has taken longer to resolve than anticipated, so I'll keep working on the run-recipes-from-firebase fixes when I return to town next week.

Once we all agree on the solution for database handler initiation and all related PRs are merged, I'll resume work on uploading remote recipes to S3 and storing metadata to firebase. Thanks!

@rugeli rugeli requested review from meganrm and mogres August 29, 2023 22:58
simulariumHelper.open_in_simularium(url)

@staticmethod
def store_results_to_s3(file_path):
Copy link
Member

@meganrm meganrm Sep 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might consider moving more of this into the handler. so in this file all you have to do is call url = AWSHandler(OPTIONS).save_file(file_path) and everything is handled by the other file

Copy link
Collaborator

@mogres mogres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! I was able to run the specified recipe and it opened up results in a browser tab.

try:
url = simulariumHelper.store_results_to_s3(simularium_file)
except Exception as e:
aws_readme_url = "https://github.com/mesoscope/cellpack/blob/feature/main/README.md#aws-s3"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is helpful, thanks for adding it in!

@rugeli rugeli merged commit b6fc051 into main Sep 22, 2023
@rugeli rugeli deleted the feature/upload-local-recipes-to-S3 branch September 22, 2023 21:18
@rugeli rugeli mentioned this pull request Oct 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants