Create organized starfish.spacetx bucket

[shanaxel42]

We need to define how we want to organize both the published datasets @dganguli will be working with. And the spaceTX datasets and results for the benchmarking.

[ambrosejcarr]

I think #1287 is relevant to the discussion of the formatted/processed separation.

[ttung]

My proposal:

Top level: A list of datasets. NOT ASSAYS.
Second level: {raw, spacetx-formatted, outputs}.
Each second-level folder has timestamped folders.

[ambrosejcarr]

Makes sense. Do we want a naming convention for the datasets? What information should be encoded in the name?

Hoping to avoid things like:

ISS_1
ISS_ambrose
ISS_deep
...

[ttung]

I would say something more about the tissue type and chemistry?

[shanaxel42]

proposal!

starfish.data.published
        assays
            datasets (some sort of schema for naming tissue type and chemistry)
                raw
                       raw_data
                formatted
                    date
                        experiment.json
                processed
                    date
                        decoded.csv (or whatever)
starfish.data.unpublished
        assays
            datasets (some sort of schema for naming tissue type and chemistry)
                raw
                       raw_data
                formatted
                    date
                        experiment.json
                processed
                    date
                        decoded.csv (or whatever)
starfish.data.spaceTX
        assays
            datasets (some sort of schema for naming tissue type and chemistry)
                raw
                       raw_data
                formatted
                    date
                        experiment.json
                processed
                    date
                        decoded.csv (or whatever)

we hand off everything underneath the spacteTX directory

[ttung]

I'm generally supportive of this approach.

What's with the top-level starfish thing?

[shanaxel42]

What's with the top-level starfish thing?

oh thats just whatever the top level is so I guess now its starfish.data.public but could be whatever I don't really have a preference for that

[ambrosejcarr]

I am downloading some slide-seq data to work with. Based on the above proposal, I intend to put it in:

starfish.data.public/published/rodriques_science_2019_slide-seq_perkinje-cerebellum/20190605/<data>

The "datasets" corresponds to <first-author-last-name>_<science>_<year>_<assay_type>_<tissue-type>

How does this sound for the "datasets" schema?

[shanaxel42]

scheme sounds fine but it should go in starfish.data.published, not starfish.data.public, the former is the new bucket

[shanaxel42]

starfish.data.spacetx/ now contains all the current spaceTX data and results we have in an organized structure. The structure as well as the original locations used to copy the data over from is described here:

ISS_30
	mouse		
		formatted: 
			https://console.aws.amazon.com/s3/buckets/spacetx.starfish.data.upload/xiaoyan_qian/

	human
		formatted: 
			https://console.aws.amazon.com/s3/buckets/spacetx.starfish.data.upload/xiaoyan_qian/ISS_human_HCA_07_MultiFOV/
			main_files/?region=us-east-1&tab=overview

		starish_results: 
			https://console.aws.amazon.com/s3/buckets/spacetx.starfish.data.upload/xiaoyan_qian/ISS_human_HCA_07_MultiFOV/
			main_files/*iss_spacetx_*

ISS_120 
	mouse		
		formatted: 
			https://console.aws.amazon.com/s3/object/spacetx.starfish.data.upload/xiaoyan_qian/
			ISS_m_brain_03/README.txt?region=us-east-1&tab=overview

	human

		formatted: 
			https://console.aws.amazon.com/s3/buckets/spacetx.starfish.data.upload/xiaoyan_qian/	
			ISS_h_brain_03/?region=us-east-1&tab=overview

FISSEQ
	mouse: 
		contributer results: 
			spacetx.starfish.data.upload/samuel_inverso/20181203-mouse-71


BaristaSEQ: 
	mouse:  
		formatted: 
			https://console.aws.amazon.com/s3/buckets/spacetx.starfish.data.public/browse/formatted/20190319/baristaseq/?region=us-east-1&tab=overview

		contributer_results: 
			https://console.aws.amazon.com/s3/object/spacetx.starfish.data.upload/xiaoyin_chen/resultsandcode.zip?region=us-east-1&tab=overview

seqFISH
	mouse:
		formatted_mulitplexed: 
			https://console.aws.amazon.com/s3/buckets/spacetx.starfish.data.upload/nico_pierson/multiplexed/

		formatted_sequential:
			spacetx.starfish.data.upload/nico_pierson/sequential/

		contributer_results_mulitplexed: 
			spacetx.starfish.data.upload/nico_pierson/multiplexed/output/ 

smFISH
	mouse: 
		formatted:
			https://console.aws.amazon.com/s3/buckets/starfish.data.spacetx/smFISH/mouse/formatted/20190214/?region=us-east-1&tab=overview


spatial transcriptomics
	mouse: 
		formatted:
			Ambrose TODO