Composite codebook support #1626

shanaxel42 · 2019-10-24T00:44:53Z

Added a slicing method in codebook that allows a user to easily get a portion of the data.
Added a new SimpleLookupDecoder for non multiplexed data that just makes a table of r/ch keys to target value and decodes.
Added a few tests to demonstrate workflows with composite experiment data

codecov-io · 2019-10-24T00:58:23Z

Codecov Report

Merging #1626 into master will increase coverage by 0.27%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1626      +/-   ##
==========================================
+ Coverage   90.06%   90.33%   +0.27%     
==========================================
  Files         223      235      +12     
  Lines        8563     9323     +760     
==========================================
+ Hits         7712     8422     +710     
- Misses        851      901      +50

Impacted Files	Coverage Δ
starfish/core/types/_spot_finding_results.py	`97.61% <ø> (ø)`	⬆️
...sh/core/spots/DecodeSpots/simple_lookup_decoder.py	`100% <100%> (ø)`
starfish/core/codebook/codebook.py	`96.93% <100%> (+0.07%)`	⬆️
starfish/core/imagestack/indexing_utils.py	`95.34% <100%> (ø)`	⬆️
starfish/core/imagestack/imagestack.py	`94.17% <100%> (+0.97%)`	⬆️
starfish/core/spots/DecodeSpots/__init__.py	`100% <100%> (ø)`	⬆️
.../spots/DecodeSpots/test/test_composite_codebook.py	`100% <100%> (ø)`
starfish/core/test/test_display.py	`42.42% <0%> (-4.25%)`	⬇️
starfish/core/image/Filter/reduce.py	`97.67% <0%> (-2.33%)`	⬇️
...hology/object/binary_mask/test/test_binary_mask.py	`94.28% <0%> (ø)`
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 995db89...0643359. Read the comment docs.

ttung · 2019-10-25T19:42:08Z

starfish/core/codebook/codebook.py

+        Parameters
+        ----------
+        indexers : Mapping[Axes, Union[int, tuple]]
+            A dictionary of dim:index where index is the value or range to index the dimension


the tuple case is not super clear to me. is it like a 2-tuple with a start and end? what if you want a random eclectic set?

Changed to support lists of arbitrary values as well

ttung · 2019-10-25T22:00:45Z

starfish/core/spots/DecodeSpots/simple_lookup_decoder.py

+        lookup_table: Dict[Tuple, str] = {}
+        for target in self.codebook[Features.TARGET]:
+            for ch_label in self.codebook[Axes.CH.value]:
+                for round_label in self.codebook[Axes.ROUND.value]:
+                    if self.codebook.loc[target, round_label, ch_label]:
+                        lookup_table[(int(round_label), int(ch_label))] = str(target.values)


this seems slow, but the number of targets is probably not horrendous.

it wouldn't surprise me, however, if there were a faster way to do this.

Also, if you are going to do it the slow way, you might as well detect scenarios where multiple bit patterns decode to the same target.

This is the same code we use in Codebook.to_json for saving out the codebook. I'm not sure I understand this "you might as well detect scenarios where multiple bit patterns decode to the same target." how would that make sense for a non multiplexed assay?

If one passes in a multiplexed codebook into this decoder, it will happily decode it in probably strange and unexpected ways.

oh yea but the description says it should only be used with non multiplexed data similarly to how the decode_per_round description says it should only be used for one hot encoded data

starfish/core/spots/DecodeSpots/simple_lookup_decoder.py

ttung · 2019-10-25T22:03:40Z

starfish/core/spots/DecodeSpots/test/test_composite_codebook.py

+    Produce an Imagestack representing a composite experiment where the first 2 rounds are
+    multiplexed data and the last round is sequential data.


do you have a test where we exercise something like what @berl has (multiplexed data and sequential data mixed into one round)?

ttung

I closed the comment thread about perf, but the rest I think still apply.

ttung

see comments pls.

ttung · 2019-11-21T03:02:55Z

starfish/core/codebook/codebook.py

+
+        Parameters
+        ----------
+        indexers : Mapping[Axes, Union[int, tuple]]


Suggested change

indexers : Mapping[Axes, Union[int, tuple]]

indexers : Mapping[Axes, Union[int, Sequence]]

(sequence encompasses both tuple and list)

ttung · 2019-11-21T03:03:05Z

starfish/core/codebook/codebook.py

@@ -389,6 +389,20 @@ def open_json(

        return cls.from_code_array(codebook_doc[DocumentKeys.MAPPINGS_KEY], n_round, n_channel)

+    def get_partial(self, indexers: Mapping[Axes, Union[int, tuple, list]]):


Suggested change

def get_partial(self, indexers: Mapping[Axes, Union[int, tuple, list]]):

def get_partial(self, indexers: Mapping[Axes, Union[int, Sequence]]):

ttung · 2019-11-21T03:03:24Z

starfish/core/imagestack/imagestack.py

-        indexers : Mapping[Axes, Union[int, tuple]]
-            A dictionary of dim:index where index is the value or range to index the dimension
+        indexers : Mapping[Axes, Union[int, tuple, list]]
+            A dictionary of dim:index where index is the value, values or range to index the


Suggested change

A dictionary of dim:index where index is the value, values or range to index the

A dictionary of dim:index where index is the value, values, or range to index the

ttung · 2019-11-21T03:03:33Z

starfish/core/imagestack/imagestack.py

        """Given a dictionary mapping the index name to either a value or a range represented as a
        tuple, return an Imagestack with each dimension indexed by position accordingly

        Parameters
        ----------
        indexers : Dict[Axes, (int/tuple)]
-            A dictionary of dim:index where index is the value or range to index the dimension
+            A dictionary of dim:index where index is the value, values or range to index the


Suggested change

A dictionary of dim:index where index is the value, values or range to index the

A dictionary of dim:index where index is the value, values, or range to index the

ttung · 2019-11-21T03:03:48Z

starfish/core/imagestack/imagestack.py

        """Given a dictionary mapping the index name to either a value or a range represented as a
        tuple, return an Imagestack with each dimension indexed accordingly

        Parameters
        ----------
-        indexers : Mapping[Axes, Union[int, tuple]]
-            A dictionary of dim:index where index is the value or range to index the dimension
+        indexers : Mapping[Axes, Union[int, tuple, list]]


Suggested change

indexers : Mapping[Axes, Union[int, tuple, list]]

indexers : Mapping[Axes, Union[int, Sequence]]

ttung · 2019-11-21T03:04:55Z

starfish/core/imagestack/indexing_utils.py

@@ -8,7 +8,8 @@


 def convert_to_selector(
-        indexers: Mapping[Axes, Union[int, slice, tuple]]) -> Mapping[Hashable, Union[int, slice]]:
+        indexers: Mapping[Axes, Union[int, slice, tuple, list]]


Suggested change

indexers: Mapping[Axes, Union[int, slice, tuple, list]]

indexers: Mapping[Axes, Union[int, slice, Sequence]]

ttung · 2019-11-21T03:05:35Z

starfish/core/imagestack/indexing_utils.py

@@ -8,7 +8,8 @@


 def convert_to_selector(
-        indexers: Mapping[Axes, Union[int, slice, tuple]]) -> Mapping[Hashable, Union[int, slice]]:
+        indexers: Mapping[Axes, Union[int, slice, tuple, list]]
+) -> Mapping[Hashable, Union[int, slice, list]]:
    """Converts a mapping of Axis to int, slice, or tuple to a mapping of str to int or slice.  The


Suggested change

"""Converts a mapping of Axis to int, slice, or tuple to a mapping of str to int or slice. The

"""Converts a mapping of Axis to int, slice, or Sequence to a mapping of str to int, slice, or list. The

ttung · 2019-11-21T03:06:19Z

starfish/core/imagestack/indexing_utils.py

-    return_dict: MutableMapping[Hashable, Union[int, slice]] = {
-        ind.value: slice(None, None) for ind in Axes}
+    return_dict: MutableMapping[Hashable, Union[int, slice, list]] = {
+        ind.value: ind.value if type(ind.value) is list


Suggested change

ind.value: ind.value if type(ind.value) is list

ind.value: ind.value if isinstance(ind.value, list)

ttung · 2019-11-21T06:01:37Z

starfish/core/spots/DecodeSpots/test/test_composite_codebook.py

+    assert np.array_equal(decoded_sequential[Features.TARGET].values, ['GENE_C', 'GENE_D'])
+
+
+def test_composite_codebook_decoding_seperate_sized_stacks():


It would be good if this had a short description like test_composite_codebook_decoding.

ttung · 2019-11-21T06:01:44Z

starfish/core/spots/DecodeSpots/test/test_composite_codebook.py

+                          ['GENE_C', 'GENE_D', 'GENE_E'])
+
+
+def test_composite_codebook_mixed_round():


shanaxel42 requested a review from ttung October 24, 2019 00:47

shanaxel42 force-pushed the saxelrod-codebook-indexing branch from 2c1be1a to 00e8ed7 Compare October 24, 2019 16:34

ttung reviewed Oct 25, 2019

View reviewed changes

ttung requested changes Nov 15, 2019

View reviewed changes

shanaxel42 force-pushed the saxelrod-codebook-indexing branch 2 times, most recently from 77e9cd3 to 450bb81 Compare November 20, 2019 01:43

shanaxel42 requested a review from ttung November 20, 2019 01:43

ttung approved these changes Nov 21, 2019

View reviewed changes

adding convience method for sliceing codebook data

0643359

shanaxel42 force-pushed the saxelrod-codebook-indexing branch from 450bb81 to 0643359 Compare November 21, 2019 18:07

shanaxel42 merged commit 6c3bf46 into master Nov 21, 2019

neuromusic mentioned this pull request Nov 27, 2019

Starfish should accept codebooks uneven number of channels across rounds #1483

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Composite codebook support #1626

Composite codebook support #1626

shanaxel42 commented Oct 24, 2019

codecov-io commented Oct 24, 2019 •

edited

Loading

ttung Oct 25, 2019

shanaxel42 Nov 20, 2019

ttung Oct 25, 2019

shanaxel42 Oct 25, 2019

ttung Oct 25, 2019

shanaxel42 Nov 4, 2019

ttung Oct 25, 2019

shanaxel42 Nov 20, 2019

ttung left a comment

ttung left a comment

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

ttung Nov 21, 2019

		Produce an Imagestack representing a composite experiment where the first 2 rounds are
		multiplexed data and the last round is sequential data.

	indexers : Mapping[Axes, Union[int, tuple]]
	indexers : Mapping[Axes, Union[int, Sequence]]

		@@ -389,6 +389,20 @@ def open_json(

		return cls.from_code_array(codebook_doc[DocumentKeys.MAPPINGS_KEY], n_round, n_channel)

		def get_partial(self, indexers: Mapping[Axes, Union[int, tuple, list]]):

	A dictionary of dim:index where index is the value, values or range to index the
	A dictionary of dim:index where index is the value, values, or range to index the

	indexers: Mapping[Axes, Union[int, slice, tuple, list]]
	indexers: Mapping[Axes, Union[int, slice, Sequence]]

	"""Converts a mapping of Axis to int, slice, or tuple to a mapping of str to int or slice. The
	"""Converts a mapping of Axis to int, slice, or Sequence to a mapping of str to int, slice, or list. The

	ind.value: ind.value if type(ind.value) is list
	ind.value: ind.value if isinstance(ind.value, list)

		assert np.array_equal(decoded_sequential[Features.TARGET].values, ['GENE_C', 'GENE_D'])


		def test_composite_codebook_decoding_seperate_sized_stacks():

		['GENE_C', 'GENE_D', 'GENE_E'])


		def test_composite_codebook_mixed_round():

Composite codebook support #1626

Composite codebook support #1626

Conversation

shanaxel42 commented Oct 24, 2019

codecov-io commented Oct 24, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttung left a comment

Choose a reason for hiding this comment

ttung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Oct 24, 2019 •

edited

Loading