Creating a standard starfish.wdl that can be run with any recipe file #1364

shanaxel42 · 2019-05-17T23:41:28Z

Created a standard starfish.wdl file that can be used to process per FOV. It takes in as inputs
-path to experiment.json
-The number of fovs in the experiment
-the url path to a python recipe file

I think the easiest way to have user defined pipelines is to just allow them to upload a recipe file to their own repos, this way they don't need to make a PR to starfish to get a recipe in. Any recipe file just needs to implement a method called process_fov(field_num: int, experiement_str: str) that processes a single fov and returns a decoded intensity table. The starfish.wdl parallelizes per fov, downloads whatever user defined recipe file as recipe.py, imports it as a module called recipe and runs recipe.process_fov(field_num: int, experiement_str: str) it then saves each decoded intensity table and concatenates them all into decoded_concatenated.csv.

There's also three examples for the ISS spaceTX, ISS published, and merfish published pipelines.

I also deleted the outdated old wdl directory.

fixes: #1280 but probs needs some more documentation.

codecov-io · 2019-05-17T23:49:50Z

Codecov Report

Merging #1364 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1364   +/-   ##
=======================================
  Coverage   89.05%   89.05%           
=======================================
  Files         147      147           
  Lines        5355     5355           
=======================================
  Hits         4769     4769           
  Misses        586      586

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30357c1...b1d76f9. Read the comment docs.

ambrosejcarr

Great ideas! I've requested a few changes, but it should be fine to merge afterwards. This will need to be refactored when we successfully figure out per-round processing, but I think that PR-to-be is the right place to do that work. 👍

ambrosejcarr · 2019-05-18T19:52:39Z

workflows/wdl/iss_published/recipe.py

+from starfish.types import Axes
+
+
+def process_fov(field_num: int, experiement_str: str):


Suggested change

def process_fov(field_num: int, experiement_str: str):

def process_fov(field_num: int, experiment_str: str):

"""Process a single field of view of ISS data

Parameters

----------

field_num : int

the field of view to process

experiment_str : int

path of experiment json file

Returns

-------

DecodedSpots :

tabular object containing the locations of detected spots.

"""

ambrosejcarr · 2019-05-18T19:52:52Z

workflows/wdl/iss_published/recipe.py

+    fov_str: str = f"fov_{int(field_num):03d}"
+
+    # load experiment
+    experiment = starfish.Experiment.from_json(experiement_str)


Suggested change

experiment = starfish.Experiment.from_json(experiement_str)

experiment = starfish.Experiment.from_json(experiment_str)

ambrosejcarr · 2019-05-18T19:58:23Z

workflows/wdl/iss_spaceTX/recipe.py

+from starfish.types import Axes
+
+
+def process_fov(field_num: int, experiement_str: str):


Suggested change

def process_fov(field_num: int, experiement_str: str):

def process_fov(field_num: int, experiment_str: str):

"""Process a single field of view of ISS data

Parameters

----------

field_num : int

the field of view to process

experiment_str : int

path of experiment json file

Returns

-------

DecodedSpots :

tabular object containing the locations of detected spots.

"""

ambrosejcarr · 2019-05-18T19:59:05Z

workflows/wdl/iss_spaceTX/recipe.py

+    fov_str: str = f"fov_{int(field_num):03d}"
+
+    # load experiment
+    experiment = starfish.Experiment.from_json(experiement_str)


Suggested change

experiment = starfish.Experiment.from_json(experiement_str)

experiment = starfish.Experiment.from_json(experiment_str)

ambrosejcarr · 2019-05-18T19:59:45Z

workflows/wdl/merfish_published/recipe.py

+    # load experiment
+    experiment = starfish.Experiment.from_json(experiement_str)
+
+    print(f"loading fov: {fov_str}")


Suggested change

print(f"loading fov: {fov_str}")

print(f"Loading fov: {fov_str}")

Other recipes had upper-case print outs.

ambrosejcarr · 2019-05-18T20:00:37Z

workflows/wdl/merfish_published/recipe.py

+    ghp = Filter.GaussianHighPass(sigma=3)
+    high_passed = ghp.run(imgs, verbose=True, in_place=False)
+
+    print("deconvoling")


Suggested change

print("deconvoling")

print("Deconvolve")

ambrosejcarr · 2019-05-18T20:00:55Z

workflows/wdl/merfish_published/recipe.py

+    dpsf = Filter.DeconvolvePSF(num_iter=15, sigma=2, clip_method=Clip.SCALE_BY_CHUNK)
+    deconvolved = dpsf.run(high_passed, verbose=True, in_place=False)
+
+    print("guassian low pass")


Suggested change

print("guassian low pass")

print("Gaussian Low Pass")

ambrosejcarr · 2019-05-18T20:01:16Z

workflows/wdl/merfish_published/recipe.py

+        scaled = data / scale_factors[selector[Axes.ROUND.value], selector[Axes.CH.value]]
+        filtered_imgs.set_slice(selector, scaled, [Axes.ZPLANE])
+
+    print("decoding")


Suggested change

print("decoding")

print("Decode")

ambrosejcarr · 2019-05-18T20:03:04Z

workflows/wdl/starfish.wdl

+
+    String experiment
+    Int field_of_view
+    String recipe_file


Suggested change

String recipe_file

File recipe_file

If you declare this a file, the WDL runner will localize it for you, then you can delete the wget line below.

This does not work. The file has to be local already https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#task-inputs

ambrosejcarr · 2019-05-18T20:03:16Z

workflows/wdl/starfish.wdl

+    String recipe_file
+
+    command <<<
+        wget -O recipe.py ${recipe_file}


Suggested change

wget -O recipe.py ${recipe_file}

ttung · 2019-05-20T21:11:53Z

workflows/wdl/iss_published/recipe.py

+
+    masking_radius = 15
+    print("Filter WhiteTophat")
+    filt = Filter.WhiteTophat(masking_radius, is_volume=False)


I'd just do:

Suggested change

filt = Filter.WhiteTophat(masking_radius, is_volume=False)

filt = Filter.WhiteTophat(masking_radius=15, is_volume=False)

ttung · 2019-05-20T21:13:57Z

workflows/wdl/iss_published/recipe.py

+    print("Filter WhiteTophat")
+    filt = Filter.WhiteTophat(masking_radius, is_volume=False)
+
+    filtered_imgs = filt.run(imgs, verbose=True, in_place=False)


Consider moving the registration to before filtering so you can do inplace processing.

ttung · 2019-05-20T21:16:42Z

workflows/wdl/iss_published/recipe.py

+    registered_imgs = warp.run(filtered_imgs, transforms_list=transforms_list, in_place=False, verbose=True)
+
+    print("Detecting")
+    p = DetectSpots.BlobDetector(


Suggested change

p = DetectSpots.BlobDetector(

detector = DetectSpots.BlobDetector(

ttung · 2019-05-20T21:16:51Z

workflows/wdl/iss_published/recipe.py

+
+    decoded = experiment.codebook.decode_per_round_max(intensities)
+    df = decoded.to_decoded_spots()
+    return df


ttung · 2019-05-20T21:18:02Z

workflows/wdl/iss_spaceTX/recipe.py

+
+    # find threshold
+    tmp = dots.sel({Axes.ROUND:0, Axes.CH:0, Axes.ZPLANE:0})
+    dots_threshold = np.percentile(np.ravel(tmp.xarray.values), 50)


this is incompatible with the pipeline architecture. do we have an issue for fixing this?

ttung · 2019-05-20T21:19:32Z

workflows/wdl/merfish_published/recipe.py

+    glp = Filter.GaussianLowPass(sigma=1)
+    low_passed = glp.run(deconvolved, in_place=False, verbose=True)
+
+    scale_factors = {


this is also incompatible with the pipeline infrastructure.

ttung · 2019-05-20T21:19:46Z

workflows/wdl/merfish_published/recipe.py

+
+    spot_intensities = initial_spot_intensities.loc[initial_spot_intensities[Features.PASSES_THRESHOLDS]]
+    df = spot_intensities.to_decoded_spots()
+    return df


nl after this.

ttung · 2019-05-20T21:21:15Z

workflows/wdl/starfish.wdl

+
+    command <<<
+        python <<CODE
+        files = "${sep=' ' decoded_csvs}".strip().split()


it's kind of concerning that this is a bunch of python code floating in the ether.

Yea I thought about moving this in starfish and making a wdl_utils file? But I thought that might be kind of random if it was the only wdl related thing...down to do that though if you think it makes more sense

Shannon Axelrod added 2 commits May 17, 2019 16:38

new branch with standard wdl files

003b889

changing to master paths

1e14c3d

updating method signature

74acdae

shanaxel42 requested a review from ambrosejcarr May 18, 2019 00:03

Shannon Axelrod added 3 commits May 17, 2019 17:05

changing location for testing

47b4c2c

testing

f60c6ec

fixing big

c5caa7b

ambrosejcarr approved these changes May 18, 2019

View reviewed changes

review changes

b1d76f9

shanaxel42 requested a review from ttung May 20, 2019 17:44

shanaxel42 merged commit 445e40a into master May 20, 2019

shanaxel42 deleted the saxelrod-standard-wdl branch May 20, 2019 20:33

ttung reviewed May 20, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating a standard starfish.wdl that can be run with any recipe file #1364

Creating a standard starfish.wdl that can be run with any recipe file #1364

shanaxel42 commented May 17, 2019 •

edited

Loading

codecov-io commented May 17, 2019 •

edited

Loading

ambrosejcarr left a comment •

edited

Loading

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

ambrosejcarr May 18, 2019

shanaxel42 May 20, 2019

ambrosejcarr May 18, 2019

ttung May 20, 2019

ttung May 20, 2019

ttung May 20, 2019

ttung May 20, 2019

ttung May 20, 2019

ttung May 20, 2019

ttung May 20, 2019

ttung May 20, 2019

shanaxel42 May 20, 2019

		from starfish.types import Axes


		def process_fov(field_num: int, experiement_str: str):

-def process_fov(field_num: int, experiement_str: str):
+def process_fov(field_num: int, experiment_str: str):
+    """Process a single field of view of ISS data
+    Parameters
+    ----------
+    field_num : int
+        the field of view to process
+    experiment_str : int
+        path of experiment json file
+    Returns
+    -------
+    DecodedSpots :
+        tabular object containing the locations of detected spots.
+    """

	experiment = starfish.Experiment.from_json(experiement_str)
	experiment = starfish.Experiment.from_json(experiment_str)

	print(f"loading fov: {fov_str}")
	print(f"Loading fov: {fov_str}")

	filt = Filter.WhiteTophat(masking_radius, is_volume=False)
	filt = Filter.WhiteTophat(masking_radius=15, is_volume=False)

	p = DetectSpots.BlobDetector(
	detector = DetectSpots.BlobDetector(

Creating a standard starfish.wdl that can be run with any recipe file #1364

Creating a standard starfish.wdl that can be run with any recipe file #1364

Conversation

shanaxel42 commented May 17, 2019 • edited Loading

codecov-io commented May 17, 2019 • edited Loading

Codecov Report

ambrosejcarr left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanaxel42 commented May 17, 2019 •

edited

Loading

codecov-io commented May 17, 2019 •

edited

Loading

ambrosejcarr left a comment •

edited

Loading