update readme

README.md

@@ -557,6 +557,41 @@ classifier = LinearClassifier(128, 10)
 When you reach the limits of the flexibility provided by Flash, then seamlessly transition to PyTorch Lightning which
 gives you the most flexibility because it is simply organized PyTorch.
 
+## Visualization
+
+Predictions from image and video tasks can be visualized through an [integration with FiftyOne](https://lightning-flash.readthedocs.io/en/latest/integrations/fiftyone.html), allowing you to better understand and analyze how your model is performing.
+
+```python
+from flash.core.data.utils import download_data
+from flash.core.integrations.fiftyone import visualize
+from flash.image import ObjectDetector
+from flash.image.detection.serialization import FiftyOneDetectionLabels
+
+# 1. Download the data
+# Dataset Credit: https://www.kaggle.com/ultralytics/coco128
+download_data(
+    "https://github.com/zhiqwang/yolov5-rt-stack/releases/download/v0.3.0/coco128.zip",
+    "data/",
+)
+
+# 2. Load the model from a checkpoint and use the FiftyOne serializer
+model = ObjectDetector.load_from_checkpoint(
+    "https://flash-weights.s3.amazonaws.com/object_detection_model.pt"
+)
+model.serializer = FiftyOneDetectionLabels()
+
+# 3. Detect the object on the images
+filepaths = [
+    "data/coco128/images/train2017/000000000025.jpg",
+    "data/coco128/images/train2017/000000000520.jpg",
+    "data/coco128/images/train2017/000000000532.jpg",
+]
+predictions = model.predict(filepaths)
+
+# 4. Visualize predictions in FiftyOne App
+session = visualize(predictions, filepaths=filepaths)
+```
+
 ## Contribute!
 The lightning + Flash team is hard at work building more tasks for common deep-learning use cases. But we're looking for incredible contributors like you to submit new tasks!
 

docs/source/index.rst

@@ -70,7 +70,6 @@ Lightning Flash
 
    integrations/fiftyone
 
-
 .. toctree::
    :maxdepth: 1
    :caption: Contributing a Task

docs/source/integrations/fiftyone.rst

@@ -27,14 +27,13 @@ are supported!
 Installation
 ************
 
-In order to utilize this integration with FiftyOne, you will need to
-:ref:`install the tool<fiftyone:installing-fiftyone>`:
+In order to utilize this integration, you will need to
+:ref:`install FiftyOne <fiftyone:installing-fiftyone>`:
 
 .. code:: shell
 
     pip install fiftyone
 
-
 *****************************
 Visualizing Flash predictions
 *****************************
@@ -52,14 +51,30 @@ one of the following serializers:
 The :func:`~flash.core.integrations.fiftyone.visualize` function then lets you visualize
 your predictions in the
 :ref:`FiftyOne App <fiftyone:fiftyone-app>`. This function accepts a list of
-dictionaries containing :ref:`FiftyOne Label<fiftyone:using-labels>` objects
-and filepaths which is the exact output of the FiftyOne serializers when the flag
-``return_filepath=True`` is specified.
+dictionaries containing :ref:`FiftyOne Label <fiftyone:using-labels>` objects
+and filepaths, which is exactly the output of the FiftyOne serializers when the
+``return_filepath=True`` option is specified.
 
 .. literalinclude:: ../../../flash_examples/integrations/fiftyone/image_classification.py
     :language: python
     :lines: 14-
 
+The :func:`~flash.core.integrations.fiftyone.visualize` function can be used in
+all of the following environments:
+
+-   **Local Python shell**: The App will launch in a new tab in your default
+    web browser
+-   **Remote Python shell**: Pass the ``remote=True`` option and then follow
+    the instructions printed to your remote shell to open the App in your
+    browser on your local machine
+-   **Jupyter notebook**: The App will launch in the output of your current
+    cell
+-   **Google Colab**: The App will launch in the output of your current cell
+-   **Python script**: Pass the ``wait=True`` option to block execution of your
+    script until the App is closed
+
+See :ref:`this page <fiftyone:environments>` for more information about
+using the FiftyOne App in different environments.
 
 ***********************
 Using FiftyOne datasets
@@ -73,15 +88,14 @@ ground truth annotations. This allows you to perform more complex analysis with
 
 The
 :meth:`~flash.core.data.data_module.DataModule.from_fiftyone`
-method allows you to load your FiftyOne Datasets directly into a
+method allows you to load your FiftyOne datasets directly into a
 :class:`~flash.core.data.data_module.DataModule` to be used for training,
 testing, or inference.
 
 .. literalinclude:: ../../../flash_examples/integrations/fiftyone/image_classification_fiftyone_datasets.py
     :language: python
     :lines: 14-
 
-
 **********************
 Visualizing embeddings
 **********************
@@ -130,6 +144,7 @@ FiftyOneSegmentationLabels
 
 .. autoclass:: flash.image.segmentation.serialization.FiftyOneSegmentationLabels
     :members:
+    :noindex:
 
 .. _fiftyone_detection_labels:
 

flash/core/integrations/fiftyone/utils.py

@@ -20,51 +20,72 @@
 
 
 def visualize(
-    labels: Union[List[Label], List[Dict[str, Label]]],
+    predictions: Union[List[Label], List[Dict[str, Label]]],
     filepaths: Optional[List[str]] = None,
-    wait: Optional[bool] = True,
     label_field: Optional[str] = "predictions",
+    wait: Optional[bool] = False,
     **kwargs
 ) -> Optional[Session]:
-    """Use the result of a FiftyOne serializer to visualize predictions in the
-    FiftyOne App.
+    """Visualizes predictions from a model with a FiftyOne Serializer in the
+    :ref:`FiftyOne App <fiftyone:fiftyone-app>`.
+
+    This method can be used in all of the following environments:
+
+    -   **Local Python shell**: The App will launch in a new tab in your
+        default web browser.
+    -   **Remote Python shell**: Pass the ``remote=True`` option to this method
+        and then follow the instructions printed to your remote shell to open
+        the App in your browser on your local machine.
+    -   **Jupyter notebook**: The App will launch in the output of your current
+        cell.
+    -   **Google Colab**: The App will launch in the output of your current
+        cell.
+    -   **Python script**: Pass the ``wait=True`` option to block execution of
+        your script until the App is closed.
+
+    See :ref:`this page <fiftyone:environments>` for more information about
+    using the FiftyOne App in different environments.
 
     Args:
-        labels: Either a list of FiftyOne labels that will be applied to the
-            corresponding filepaths provided with through `filepath` or
-            `datamodule`. Or a list of dictionaries containing image/video
-            filepaths and corresponding FiftyOne labels.
+        predictions: Can be either a list of FiftyOne labels that will be
+            matched with the corresponding ``filepaths``, or a list of
+            dictionaries with "filepath" and "predictions" keys that contains
+            the filepaths and predictions.
         filepaths: A list of filepaths to images or videos corresponding to the
-            provided `labels`.
-        wait: A boolean determining whether to launch the FiftyOne session and
-            wait until the session is closed or whether to return immediately.
-        label_field: The string of the label field in the FiftyOne dataset
-            containing predictions
+            provided ``predictions``.
+        label_field: The name of the label field in which to store the
+            predictions in the FiftyOne dataset.
+        wait: Whether to block execution until the FiftyOne App is closed.
+        **kwargs: Optional keyword arguments for
+            :meth:`fiftyone:fiftyone.core.session.launch_app`.
+
+    Returns:
+        a :class:`fiftyone:fiftyone.core.session.Session`
     """
     if not _FIFTYONE_AVAILABLE:
         raise ModuleNotFoundError("Please, `pip install fiftyone`.")
     if flash._IS_TESTING:
         return None
 
     # Flatten list if batches were used
-    if all(isinstance(fl, list) for fl in labels):
-        labels = list(chain.from_iterable(labels))
+    if all(isinstance(fl, list) for fl in predictions):
+        predictions = list(chain.from_iterable(predictions))
 
-    if all(isinstance(fl, dict) for fl in labels):
-        filepaths = [lab["filepath"] for lab in labels]
-        labels = [lab["predictions"] for lab in labels]
+    if all(isinstance(fl, dict) for fl in predictions):
+        filepaths = [lab["filepath"] for lab in predictions]
+        labels = [lab["predictions"] for lab in predictions]
+    else:
+        labels = predictions
 
     if filepaths is None:
         raise ValueError("The `filepaths` argument is required if filepaths are not provided in `labels`.")
 
     dataset = fo.Dataset()
     if filepaths:
-        dataset.add_labeled_images(
-            list(zip(filepaths, labels)),
-            LabeledImageTupleSampleParser(),
-            label_field=label_field,
-        )
+        dataset.add_samples([fo.Sample(filepath=f, **{label_field: l}) for f, l in zip(filepaths, labels)])
+
     session = fo.launch_app(dataset, **kwargs)
     if wait:
         session.wait()
+
     return session

flash/image/segmentation/serialization.py

@@ -39,15 +39,16 @@
 
 
 class SegmentationLabels(Serializer):
+    """A :class:`.Serializer` which converts the model outputs to the label of
+    the argmax classification per pixel in the image for semantic segmentation
+    tasks.
 
-    def __init__(self, labels_map: Optional[Dict[int, Tuple[int, int, int]]] = None, visualize: bool = False):
-        """A :class:`.Serializer` which converts the model outputs to the label of the argmax classification
-        per pixel in the image for semantic segmentation tasks.
+    Args:
+        labels_map: A dictionary that map the labels ids to pixel intensities.
+        visualize: Wether to visualize the image labels.
+    """
 
-        Args:
-            labels_map: A dictionary that map the labels ids to pixel intensities.
-            visualize: Wether to visualize the image labels.
-        """
+    def __init__(self, labels_map: Optional[Dict[int, Tuple[int, int, int]]] = None, visualize: bool = False):
         super().__init__()
         self.labels_map = labels_map
         self.visualize = visualize
@@ -89,22 +90,23 @@ def serialize(self, sample: Dict[str, torch.Tensor]) -> torch.Tensor:
 
 
 class FiftyOneSegmentationLabels(SegmentationLabels):
+    """A :class:`.Serializer` which converts the model outputs to FiftyOne
+    segmentation format.
+
+    Args:
+        labels_map: A dictionary that map the labels ids to pixel intensities.
+        visualize: whether to visualize the image labels.
+        return_filepath: Boolean determining whether to return a dict
+            containing filepath and FiftyOne labels (True) or only a list of
+            FiftyOne labels (False).
+    """
 
     def __init__(
         self,
         labels_map: Optional[Dict[int, Tuple[int, int, int]]] = None,
         visualize: bool = False,
         return_filepath: bool = False,
     ):
-        """A :class:`.Serializer` which converts the model outputs to FiftyOne segmentation format.
-
-        Args:
-            labels_map: A dictionary that map the labels ids to pixel intensities.
-            visualize: Wether to visualize the image labels.
-            return_filepath: Boolean determining whether to return a dict
-                containing filepath and FiftyOne labels (True) or only a
-                list of FiftyOne labels (False)
-        """
         if not _FIFTYONE_AVAILABLE:
             raise ModuleNotFoundError("Please, run `pip install fiftyone`.")
 

flash_examples/integrations/fiftyone/image_classification.py

@@ -51,11 +51,10 @@
 
 # 4 Predict from checkpoint
 model = ImageClassifier.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/image_classification_model.pt")
-model.serializer = FiftyOneLabels(return_filepath=True)
+model.serializer = FiftyOneLabels(return_filepath=True)  # output FiftyOne format
 predictions = trainer.predict(model, datamodule=datamodule)
-
 predictions = list(chain.from_iterable(predictions))  # flatten batches
 
-# 5. Visualize predictions in FiftyOne
-# Note: this blocks until the FiftyOne App is closed
+# 5 Visualize predictions in FiftyOne App
+# Optional: pass `wait=True` to block execution until App is closed
 session = visualize(predictions)

flash_examples/integrations/fiftyone/image_classification_fiftyone_datasets.py

@@ -39,14 +39,14 @@
     dataset_type=fo.types.ImageClassificationDirectoryTree,
 )
 
-# 3 Load FiftyOne datasets
+# 3 Load data into Flash
 datamodule = ImageClassificationData.from_fiftyone(
     train_dataset=train_dataset,
     val_dataset=val_dataset,
     test_dataset=test_dataset,
 )
 
-# 4 Fine tune a model
+# 4 Fine tune model
 model = ImageClassifier(
     backbone="resnet18",
     num_classes=datamodule.num_classes,
@@ -66,28 +66,22 @@
 
 # 5 Predict from checkpoint on data with ground truth
 model = ImageClassifier.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/image_classification_model.pt")
-model.serializer = FiftyOneLabels(return_filepath=False)
+model.serializer = FiftyOneLabels(return_filepath=False)  # output FiftyOne format
 datamodule = ImageClassificationData.from_fiftyone(predict_dataset=test_dataset)
 predictions = trainer.predict(model, datamodule=datamodule)
-
 predictions = list(chain.from_iterable(predictions))  # flatten batches
 
 # 6 Add predictions to dataset
 test_dataset.set_values("predictions", predictions)
 
-# 7 Visualize labels in the App
-session = fo.launch_app(test_dataset)
-
-# 8 Evaluate your model
-results = test_dataset.evaluate_classifications(
-    "predictions",
-    gt_field="ground_truth",
-    eval_key="eval",
-)
+# 7 Evaluate your model
+results = test_dataset.evaluate_classifications("predictions", gt_field="ground_truth", eval_key="eval")
 results.print_report()
 plot = results.plot_confusion_matrix()
 plot.show()
 
-# Only when running this in a script
-# Block until the FiftyOne App is closed
+# 8 Visualize results in the App
+session = fo.launch_app(test_dataset)
+
+# Optional: block execution until App is closed
 session.wait()

flash_examples/integrations/fiftyone/image_embedding.py

@@ -29,20 +29,17 @@
 )
 
 # 3 Load model
-embedder = ImageEmbedder(backbone="swav-imagenet", embedding_dim=128)
+embedder = ImageEmbedder(backbone="resnet101", embedding_dim=128)
 
 # 4 Generate embeddings
 filepaths = dataset.values("filepath")
 embeddings = np.stack(embedder.predict(filepaths))
 
 # 5 Visualize in FiftyOne App
 results = fob.compute_visualization(dataset, embeddings=embeddings)
-
 session = fo.launch_app(dataset)
-
 plot = results.visualize(labels="ground_truth.label")
 plot.show()
 
-# Only when running this in a script
-# Block until the FiftyOne App is closed
+# Optional: block execution until App is closed
 session.wait()