Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

neurips loader update #472

Merged
merged 8 commits into from
Feb 8, 2022
Merged

neurips loader update #472

merged 8 commits into from
Feb 8, 2022

Conversation

xlancelottx
Copy link
Contributor

Many thanks for contributing to sfaira!

PR Checklist
Please fill in the appropriate checklist below (delete whatever is not relevant). These are the most common things requested on pull requests (PRs).

  • This comment contains a description of changes (with reason)

Description of changes
Update of neurips data loader

Technical details
Both downloaded files contain multimodal data (multiome and cite), see feature types to subset

Additional context
Related discussion: #439

@davidsebfischer
Copy link
Contributor

Thanks @xlancelottx, this looks great! You'll need to run sfaira annotate once more on this data set to get meta data maps (tsv files) for ethnicity and sex as these meta data were newly added in this submission, you can leave the remaining tsvs as they are. Can you try that please?

@davidsebfischer davidsebfischer changed the base branch from dev to feature/cli_improvements February 8, 2022 10:36
@davidsebfischer
Copy link
Contributor

I moved the loader to GEO data now as we consider that more stable.

@davidsebfischer davidsebfischer self-assigned this Feb 8, 2022
@davidsebfischer davidsebfischer linked an issue Feb 8, 2022 that may be closed by this pull request
Data: NeurIPS open problems dataset dno_doi_luecken #438
Closed
@davidsebfischer davidsebfischer merged commit 1c6d08a into feature/cli_improvements Feb 8, 2022
@davidsebfischer davidsebfischer deleted the neurips_update branch February 8, 2022 14:01
davidsebfischer added a commit that referenced this pull request Feb 9, 2022
@davidsebfischer @le-ander
@lauradmartens @xlancelottx
Improvements to CLI usability (#452)
8f4bf84 
* improved guidance through CLI pipeline through action messages and error messages
* improved curation documentation
* relabled export phase in summary
* CLI: add automatic detection of container for path handling
* CLI: adapt path checking to container usecase
* CLI: make env variables usable also without containers
* CLI: add automatic PR submission in container
* fix template creation CLI run
* switched data library, CLI and curation annotation to yaml version 1.2 (#528)
* improved curation docs
* improved guidance in sfaira create-dataloader
* improved reaction to existing loaders
* improved data loader linting in CLI
* support for count and processed data matrices in same loader, deprecates "normalization"
* support for feature type annotation, allows for ATAC and CITE data (feature_type)
* support for based genetic modification meta data (gm)
* support for basic treatment meda data (treatment)
* support for cell tracking in meta studies (source_doi)
* support for spatial data
* support for VDJ data
* support for velocity data
* support for reference genome annotation
* support for arbirtrary organisms in genome container
* added phase Pe
* Data/10.1126/science.abj4008 (#531)
* d10_1126_science_abj4008 #530
* added validate to template checking
* improvements to VDJ interface
* d10_1126_science_abe6474 (#532)
* fixed sfaira structure DOI input to annotate
* annotated GEO look up
* d10_1126_sciimmunol_abd1554 (#471)
* d10_1126_sciimmunol_abd1554
* dno_doi_luecken neurips loader update (#472)
* fixed rich printing in CLI
* added documentation of compressed and r file reading
* Fix d10_1016_j_cell_2021_01_053 to work with GEO (#469)
* added validation statement at the end of finalize
* d10_1038_s41591_020_1061_7 
* updated required meta data section in docs
* disabled default cellxgene collection meta data caching which resulted in bug when collection were updated on remote

Co-authored-by: davidsebfischer <david.seb.fischer@gmail.com>
Co-authored-by: le-ander <20015434+le-ander@users.noreply.github.com>
Co-authored-by: Laura Martens <laura.d.martens@icloud.com>
Co-authored-by: xlancelottx <33050110+xlancelottx@users.noreply.github.com>
davidsebfischer added a commit that referenced this pull request Feb 9, 2022
@davidsebfischer @le-ander
@Zethson @felix0097 @Hrovatin @soerenab @lauradmartens @xlancelottx
Preparing release 3.10 (#533)
7241ee5 
* fix NA to empty template values (#434)

Signed-off-by: zethson <lukas.heumos@posteo.net>

* added bug fix for subsetting + .X method of DistributedstoreAnndata (#431)

* Feature/trainer test paths (#437)

* made trainer paths more flexible

* do not check md5 of weighs if None (#442)

* added fix to reduce size of pickle files (#440)

* Dataset/d10 1016 j cell 2020 08 001 (#402)

Co-authored-by: davidsebfischer <david.seb.fischer@gmail.com>

* improved dask array concatenatin for .X method (#444)

* Rewrite test_store.py script (#436)

* Feature/dno doi luecken (#439)

* added data loader dno_doi_luecken
* improved CLI and documentation

* 10.1101/2021.07.19.452954 and data loader tsv update

* d10_1101_2021_07_19_452954 (10.1101/2021.07.19.452954) dataloader (#333)
* New ontology map tsv interface with one tsv per ontology (#449)
* CLI commands for cache clearing and reloading (#449)

* datal loader 10.1038/s41467-021-27619-4 (desribed in #450)

* Issue template for data loader request

* d10_1038_s41586_019_1631_3: updated annotation and moved to format 1.1 (#467)

* Rewrite batch schedule classes (#448)

* rewrote batch_schedule to work directly on the supplied indicies

* Torch data adaptor (#433)

* added torch.utils.data.IterableDataset and torch Dataset class

* added adpator unit test

* cached query of assembly name to ensembl ftp server
file query was already cached, but assembly name retrieval based on release version was not cached

* added random rounding in intercalated batch data laoder

* updated test splitting

* enabled store loading from multiple paths

* deprecated optional dependency of Dataset attributes on meta data files

* fixed d10_1038_s41586_019_1631_3

* relaxed required arguments for DistributedStoreAnndata

* added a full data set batch schedule

* feature / added method to persist dask.array of DaoStroage into memory + small bug fixes (#447)

* dask.array of DaoStore can now be persisted into memory in CSR format
* improved docstring documentation for .adataptor

* extended store docs

* refactored distributedstore to store and minor obs indexing and cart default fixes

* fixed obs indexing bug with empty single cart within multi cart

Co-authored-by: felix0097 <47145207+felix0097@users.noreply.github.com>

* bug fix for move_to_memory method of CartDask class (#475)

* D10 1038 s41467 020 15543 y (#462)

* 10.1038/s41467-020-15543-y

* Feature: Option to shuffle datasets before writing dao stores (#473)

* datasets can be shuffled now before writing dao store

* Feature: match zarr chunks for randomized_batch_schedule=True for CartDask (#476)

* random_batch_schedule for DaskCart now matches dask partitions

* rearanged imports + removed debugging assertion statement

* Create dataloader (#468)

* bug fix - added .compute for .x method in DaskCart (#479)

* Feature: Save .obs data as categorical dtype (#478)

* str obs data is now stored and processed as categorical

* fixed check for categorical dtype

* Bug-Fix: d10_1038_s41467_021_27619_4 DataLoader (#480)

* put column 3 back into var dataframe and named column feature_class

* fixed gtf interface for newer ensembl releases for genes without symbol (#484)

* add wheel to python build CI to avoid legacy setup.py install (#493)

* Feature: add shuffle buffer to DaskCart (#474)

* Refactor estimator code to support estimators for torch models (#485)

* refactored estimator code into tf.keras code and base estimator code. this prepares side-by-side keras and torch code.
* added torch losses and metrics

* Improvements to CLI usability (#452)

* improved guidance through CLI pipeline through action messages and error messages
* improved curation documentation
* relabled export phase in summary
* CLI: add automatic detection of container for path handling
* CLI: adapt path checking to container usecase
* CLI: make env variables usable also without containers
* CLI: add automatic PR submission in container
* fix template creation CLI run
* switched data library, CLI and curation annotation to yaml version 1.2 (#528)
* improved curation docs
* improved guidance in sfaira create-dataloader
* improved reaction to existing loaders
* improved data loader linting in CLI
* support for count and processed data matrices in same loader, deprecates "normalization"
* support for feature type annotation, allows for ATAC and CITE data (feature_type)
* support for based genetic modification meta data (gm)
* support for basic treatment meda data (treatment)
* support for cell tracking in meta studies (source_doi)
* support for spatial data
* support for VDJ data
* support for velocity data
* support for reference genome annotation
* support for arbirtrary organisms in genome container
* added phase Pe
* Data/10.1126/science.abj4008 (#531)
* d10_1126_science_abj4008 #530
* added validate to template checking
* improvements to VDJ interface
* d10_1126_science_abe6474 (#532)
* fixed sfaira structure DOI input to annotate
* annotated GEO look up
* d10_1126_sciimmunol_abd1554 (#471)
* d10_1126_sciimmunol_abd1554
* dno_doi_luecken neurips loader update (#472)
* fixed rich printing in CLI
* added documentation of compressed and r file reading
* Fix d10_1016_j_cell_2021_01_053 to work with GEO (#469)
* added validation statement at the end of finalize
* d10_1038_s41591_020_1061_7 
* updated required meta data section in docs
* disabled default cellxgene collection meta data caching which resulted in bug when collection were updated on remote

Co-authored-by: davidsebfischer <david.seb.fischer@gmail.com>
Co-authored-by: le-ander <20015434+le-ander@users.noreply.github.com>
Co-authored-by: Laura Martens <laura.d.martens@icloud.com>
Co-authored-by: xlancelottx <33050110+xlancelottx@users.noreply.github.com>

* added new figures (#538)

* added new figures

* activated cellxgene caching for unit tests (#539)

Co-authored-by: Leander <20015434+le-ander@users.noreply.github.com>
Co-authored-by: Lukas Heumos <lukas.heumos@posteo.net>
Co-authored-by: felix0097 <47145207+felix0097@users.noreply.github.com>
Co-authored-by: Karin Hrovatin <47607471+Hrovatin@users.noreply.github.com>
Co-authored-by: soerenab <36963673+soerenab@users.noreply.github.com>
Co-authored-by: Laura Martens <laura.d.martens@icloud.com>
Co-authored-by: xlancelottx <33050110+xlancelottx@users.noreply.github.com>
@davidsebfischer davidsebfischer mentioned this pull request Feb 9, 2022
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@davidsebfischer davidsebfischer

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Data: NeurIPS open problems dataset dno_doi_luecken
2 participants
@xlancelottx @davidsebfischer