Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset from torch to torch does not write in storage #136

Merged
merged 4 commits into from
Apr 12, 2024

Conversation

mariogmarq
Copy link
Collaborator

When creating a Dataset object from a pytorch dataset it is passed as a generator, meaning that LazyIndexable will store the whole contents of it in memory. This pull request makes a small fix so we can take advantage of the indexing capabilities of the PyTorch dataset improving memory usage.

…ataset and to back to torchvision dataset does not write and memory and only reads from underlying dataset
@mariogmarq mariogmarq requested review from xehartnort and AlArgente and removed request for xehartnort February 15, 2024 16:33
Copy link

github-actions bot commented Feb 15, 2024

Coverage

Coverage Report
FileStmtsMissCoverMissing
flex/common
   utils.py751580%135–151, 189
flex/data
   dataset.py1552882%53–59, 201, 203, 215, 223–239, 289–290, 307–308, 315, 323, 328, 335–336, 398, 414–415
   dataset_pt_utils.py29293%48, 59
   fed_data_distribution.py182199%397
   lazy_indexable.py79297%55, 89
   pluggable_datasets.py62198%27
flex/datasets
   federated_datasets.py452642%35, 39–71, 75–89, 102
   flexible_datasets.py11373%29, 46–47
flex/pool
   aggregators.py50492%55–56, 145–146
   primitives_pt.py635021%46, 61–63, 104–122, 158–170, 197–206, 235–244
   primitives_tf.py42783%74–84
TOTAL128613989% 

Tests Skipped Failures Errors Time
190 4 💤 0 ❌ 0 🔥 2m 8s ⏱️

@rbnuria rbnuria closed this Apr 12, 2024
@rbnuria rbnuria reopened this Apr 12, 2024
@mariogmarq mariogmarq requested review from rbnuria and removed request for AlArgente April 12, 2024 09:00
@rbnuria rbnuria merged commit 239292c into main Apr 12, 2024
4 checks passed
@rbnuria rbnuria deleted the Dataset-from-torch-to-torch-does-not-write-in-storage branch April 12, 2024 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants