Merge branch 'master' into feature-olci-s3

CHANGELOG.md

@@ -1,3 +1,68 @@
+## Version 0.24.0 (2020/11/16)
+
+### Issues Closed
+
+* [Issue 1412](https://github.com/pytroll/satpy/issues/1412) - Mimic reader fails when multiple times are provided to Scene object
+* [Issue 1409](https://github.com/pytroll/satpy/issues/1409) - "Unexpected number of scanlines!" when reading AVHRR GAC data
+* [Issue 1399](https://github.com/pytroll/satpy/issues/1399) - Customes Scene creation from MultiScene.from_files ([PR 1400](https://github.com/pytroll/satpy/pull/1400))
+* [Issue 1396](https://github.com/pytroll/satpy/issues/1396) - reader_kwargs should differentiate between different readers ([PR 1397](https://github.com/pytroll/satpy/pull/1397))
+* [Issue 1389](https://github.com/pytroll/satpy/issues/1389) - Can't load angle data from msi_safe in version 0.23 ([PR 1391](https://github.com/pytroll/satpy/pull/1391))
+* [Issue 1387](https://github.com/pytroll/satpy/issues/1387) - NUCAPS time format of data from CLASS ([PR 1388](https://github.com/pytroll/satpy/pull/1388))
+* [Issue 1371](https://github.com/pytroll/satpy/issues/1371) - MIMIC reader available_dataset_names returns 1d lat/lon fields ([PR 1392](https://github.com/pytroll/satpy/pull/1392))
+* [Issue 1343](https://github.com/pytroll/satpy/issues/1343) - Feature Request: available_readers to return alphabetical order
+* [Issue 1224](https://github.com/pytroll/satpy/issues/1224) - GRIB-2/ICON geolocation unknown or invalid for western hemisphere ([PR 1296](https://github.com/pytroll/satpy/pull/1296))
+
+In this release 9 issues were closed.
+
+### Pull Requests Merged
+
+#### Bugs fixed
+
+* [PR 1435](https://github.com/pytroll/satpy/pull/1435) - Fix tests for GEOFlippableFileYAMLReader after AreaDefinition.area_extent being immutable
+* [PR 1433](https://github.com/pytroll/satpy/pull/1433) - Fix cloud-free pixels in cloudtop height composite
+* [PR 1432](https://github.com/pytroll/satpy/pull/1432) - Fix enhance2dataset to support P-mode datasets
+* [PR 1431](https://github.com/pytroll/satpy/pull/1431) - Fix crash when TLE files are missing
+* [PR 1430](https://github.com/pytroll/satpy/pull/1430) - Fix infer_mode not using the band coordinate
+* [PR 1428](https://github.com/pytroll/satpy/pull/1428) - Bugfix NWC SAF GEO v2016 area definition
+* [PR 1422](https://github.com/pytroll/satpy/pull/1422) - Fix HDF5 utility file handler not decoding byte arrays consistently
+* [PR 1413](https://github.com/pytroll/satpy/pull/1413) - Fix pyspectral link in the main doc page
+* [PR 1407](https://github.com/pytroll/satpy/pull/1407) - Fix mersi 2 angles reading
+* [PR 1392](https://github.com/pytroll/satpy/pull/1392) - Remove 1-D lat/lon variables from mimic reader's available datasets ([1371](https://github.com/pytroll/satpy/issues/1371))
+* [PR 1391](https://github.com/pytroll/satpy/pull/1391) - Fix the MSI / Sentinel-2 reader so it uses new DataID ([1389](https://github.com/pytroll/satpy/issues/1389))
+* [PR 1388](https://github.com/pytroll/satpy/pull/1388) - Fix handling of new date string formats in NUCAPS reader ([1387](https://github.com/pytroll/satpy/issues/1387))
+* [PR 1382](https://github.com/pytroll/satpy/pull/1382) - Fixed bug getting name to the calibration in mitiff writer
+* [PR 1296](https://github.com/pytroll/satpy/pull/1296) - Fix grib reader handling for data on 0-360 longitude ([1224](https://github.com/pytroll/satpy/issues/1224))
+
+#### Features added
+
+* [PR 1420](https://github.com/pytroll/satpy/pull/1420) - Add support for Near-realtime VIIRS L1b data.
+* [PR 1411](https://github.com/pytroll/satpy/pull/1411) - Added MERSI-2 file pattern for data from NMSC
+* [PR 1406](https://github.com/pytroll/satpy/pull/1406) - Handle bilinear caching in Pyresample
+* [PR 1405](https://github.com/pytroll/satpy/pull/1405) - Add FIR product to seviri_l2_grib reader
+* [PR 1401](https://github.com/pytroll/satpy/pull/1401) - Add function to the SLSTR L1 reader to enable correction of VIS radiances.
+* [PR 1400](https://github.com/pytroll/satpy/pull/1400) - Improve customisation in multiscene creation ([1399](https://github.com/pytroll/satpy/issues/1399))
+* [PR 1397](https://github.com/pytroll/satpy/pull/1397) - Allow different kwargs for different readers ([1396](https://github.com/pytroll/satpy/issues/1396))
+* [PR 1394](https://github.com/pytroll/satpy/pull/1394) - Add satpy cf-reader and eumetsat gac reader ([1205](https://github.com/pytroll/satpy/issues/1205))
+* [PR 1390](https://github.com/pytroll/satpy/pull/1390) - Add support to Pyspectral NIRReflectance masking limit
+* [PR 1378](https://github.com/pytroll/satpy/pull/1378) - Alphabetize available_readers method and update documentation
+
+#### Documentation changes
+
+* [PR 1415](https://github.com/pytroll/satpy/pull/1415) - Update Code of Conduct contact email to groups.io address
+* [PR 1413](https://github.com/pytroll/satpy/pull/1413) - Fix pyspectral link in the main doc page
+* [PR 1374](https://github.com/pytroll/satpy/pull/1374) - DOC: add conda-forge badge
+
+#### Backward incompatible changes
+
+* [PR 1360](https://github.com/pytroll/satpy/pull/1360) - Create new ModifierBase class and move existing modifiers to satpy.modifiers
+
+#### Refactoring
+
+* [PR 1360](https://github.com/pytroll/satpy/pull/1360) - Create new ModifierBase class and move existing modifiers to satpy.modifiers
+
+In this release 29 pull requests were closed.
+
+
 ## Version 0.23.0 (2020/09/18)
 
 ### Issues Closed

doc/source/api/satpy.composites.rst

@@ -52,7 +52,6 @@ satpy.composites.viirs module
    :undoc-members:
    :show-inheritance:
 
-
 Module contents
 ---------------
 

doc/source/api/satpy.dataset.rst

@@ -0,0 +1,45 @@
+satpy.dataset package
+=====================
+
+Submodules
+----------
+
+satpy.dataset.anc\_vars module
+------------------------------
+
+.. automodule:: satpy.dataset.anc_vars
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+satpy.dataset.data\_dict module
+-------------------------------
+
+.. automodule:: satpy.dataset.data_dict
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+satpy.dataset.dataid module
+---------------------------
+
+.. automodule:: satpy.dataset.dataid
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+satpy.dataset.metadata module
+-----------------------------
+
+.. automodule:: satpy.dataset.metadata
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: satpy.dataset
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/satpy.enhancements.rst

@@ -28,7 +28,6 @@ satpy.enhancements.viirs module
    :undoc-members:
    :show-inheritance:
 
-
 Module contents
 ---------------
 

doc/source/api/satpy.readers.rst

@@ -60,6 +60,14 @@ satpy.readers.ahi\_hsd module
    :undoc-members:
    :show-inheritance:
 
+satpy.readers.ahi\_l1b\_gridded\_bin module
+-------------------------------------------
+
+.. automodule:: satpy.readers.ahi_l1b_gridded_bin
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 satpy.readers.ami\_l1b module
 -----------------------------
 
@@ -428,6 +436,14 @@ satpy.readers.sar\_c\_safe module
    :undoc-members:
    :show-inheritance:
 
+satpy.readers.satpy\_cf\_nc module
+----------------------------------
+
+.. automodule:: satpy.readers.satpy_cf_nc
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 satpy.readers.scatsat1\_l2b module
 ----------------------------------
 
@@ -652,7 +668,6 @@ satpy.readers.yaml\_reader module
    :undoc-members:
    :show-inheritance:
 
-
 Module contents
 ---------------
 

doc/source/api/satpy.rst

@@ -8,6 +8,7 @@ Subpackages
    :maxdepth: 4
 
    satpy.composites
+   satpy.dataset
    satpy.demo
    satpy.enhancements
    satpy.readers
@@ -24,10 +25,10 @@ satpy.config module
    :undoc-members:
    :show-inheritance:
 
-satpy.dataset module
---------------------
+satpy.dependency\_tree module
+-----------------------------
 
-.. automodule:: satpy.dataset
+.. automodule:: satpy.dependency_tree
    :members:
    :undoc-members:
    :show-inheritance:
@@ -80,7 +81,6 @@ satpy.utils module
    :undoc-members:
    :show-inheritance:
 
-
 Module contents
 ---------------
 

doc/source/api/satpy.writers.rst

@@ -60,7 +60,6 @@ satpy.writers.utils module
    :undoc-members:
    :show-inheritance:
 
-
 Module contents
 ---------------
 

doc/source/readers.rst

@@ -251,3 +251,9 @@ hdf-eos based readers
 
 .. automodule:: satpy.readers.modis_l2
     :noindex:
+
+satpy cf nc readers
+---------------------
+
+.. automodule:: satpy.readers.satpy_cf_nc
+    :noindex:

satpy/composites/__init__.py

@@ -25,11 +25,6 @@
 import numpy as np
 import xarray as xr
 
-try:
-    from yaml import UnsafeLoader
-except ImportError:
-    from yaml import Loader as UnsafeLoader
-
 from satpy.config import get_environ_ancpath
 from satpy.dataset import DataID, combine_metadata
 from satpy.dataset.dataid import minimal_default_keys_config
@@ -275,7 +270,7 @@ def infer_mode(cls, data_arr):
             return data_arr.attrs['mode']
         if 'bands' not in data_arr.dims:
             return cls.modes[1]
-        if 'bands' in data_arr.coords and isinstance(data_arr.coords['bands'][0], str):
+        if 'bands' in data_arr.coords and isinstance(data_arr.coords['bands'][0].item(), str):
             return ''.join(data_arr.coords['bands'].values)
         return cls.modes[data_arr.sizes['bands']]
 
@@ -564,12 +559,13 @@ def __call__(self, projectables, **kwargs):
         return super(DayNightCompositor, self).__call__(data, **kwargs)
 
 
-def enhance2dataset(dset):
-    """Return the enhancemened to dataset *dset* as an array."""
+def enhance2dataset(dset, convert_p=False):
+    """Return the enhancement dataset *dset* as an array.
+
+    If `convert_p` is True, enhancements generating a P mode will be converted to RGB or RGBA.
+    """
     attrs = dset.attrs
-    img = get_enhanced_image(dset)
-    # Clip image data to interval [0.0, 1.0]
-    data = img.data.clip(0.0, 1.0)
+    data = _get_data_from_enhanced_image(dset, convert_p)
     data.attrs = attrs
     # remove 'mode' if it is specified since it may have been updated
     data.attrs.pop('mode', None)
@@ -578,10 +574,31 @@ def enhance2dataset(dset):
     return data
 
 
+def _get_data_from_enhanced_image(dset, convert_p):
+    img = get_enhanced_image(dset)
+    if convert_p and img.mode == 'P':
+        img = _apply_palette_to_image(img)
+    if img.mode != 'P':
+        data = img.data.clip(0.0, 1.0)
+    else:
+        data = img.data
+    return data
+
+
+def _apply_palette_to_image(img):
+    if len(img.palette[0]) == 3:
+        img = img.convert('RGB')
+    elif len(img.palette[0]) == 4:
+        img = img.convert('RGBA')
+    return img
+
+
 def add_bands(data, bands):
     """Add bands so that they match *bands*."""
     # Add R, G and B bands, remove L band
     bands = bands.compute()
+    if 'P' in data['bands'].data or 'P' in bands.data:
+        raise NotImplementedError('Cannot mix datasets of mode P with other datasets at the moment.')
     if 'L' in data['bands'].data and 'R' in bands.data:
         lum = data.sel(bands='L')
         # Keep 'A' if it was present
@@ -1025,11 +1042,9 @@ class BackgroundCompositor(GenericCompositor):
     def __call__(self, projectables, *args, **kwargs):
         """Call the compositor."""
         projectables = self.match_data_arrays(projectables)
-
         # Get enhanced datasets
-        foreground = enhance2dataset(projectables[0])
-        background = enhance2dataset(projectables[1])
-
+        foreground = enhance2dataset(projectables[0], convert_p=True)
+        background = enhance2dataset(projectables[1], convert_p=True)
         # Adjust bands so that they match
         # L/RGB -> RGB/RGB
         # LA/RGB -> RGBA/RGBA

satpy/composites/cloud_products.py

@@ -51,17 +51,21 @@ def __call__(self, projectables, **info):
             raise ValueError("Expected 3 datasets, got %d" %
                              (len(projectables), ))
         data, palette, status = projectables
+        fill_value_color = palette.attrs.get("fill_value_color", [0, 0, 0])
         colormap, palette = self.build_colormap(palette, data.attrs)
-        channels = colormap.colorize(np.asanyarray(data))
-        not_nan = np.logical_and(self._get_mask_from_data(data), status != status.attrs['_FillValue'])
-        not_cloud_free = np.logical_or((status + 1) % 2, np.logical_not(not_nan))
-        chans = []
-        for channel in channels:
-            chan = self._create_masked_dataarray_like(channel, data, not_nan)
-            # Set cloud-free pixels as black
-            chans.append(chan.where(not_cloud_free, 0))
-
-        res = GenericCompositor.__call__(self, chans, **data.attrs)
+        mapped_channels = colormap.colorize(data.data)
+        valid = status != status.attrs['_FillValue']
+        # cloud-free pixels are marked invalid (fill_value in ctth_alti) but have status set to 1.
+        status_not_cloud_free = status % 2 == 0
+        not_cloud_free = np.logical_or(status_not_cloud_free, np.logical_not(valid))
+
+        channels = []
+        for (channel, cloud_free_color) in zip(mapped_channels, fill_value_color):
+            channel_data = self._create_masked_dataarray_like(channel, data, valid)
+            # Set cloud-free pixels as fill_value_color
+            channels.append(channel_data.where(not_cloud_free, cloud_free_color))
+
+        res = GenericCompositor.__call__(self, channels, **data.attrs)
         res.attrs['_FillValue'] = np.nan
         return res