FORCE version 3.8.00 released on January 31, 2025

[davidfrantz]

Below is an overview of implemented changes.

Most important, Sentinel-2C is supported!

Changelog:

• FORCE L1AS

  • force-level1-sentinel2 was finally removed after being deprecated for a while now.
    Use force-level1-csd. Contributions to substitute this program using the
    Copernicus Data Space Ecosystem endpoint are welcome and appreciated!

  • force-level1-csd currently suffers from a change of Google's way of indexing images.
    The previously used mechanic has been exchanged for BigQueries.
    This is not implemented yet.
    In addition, it may generate costs if it would be implemented.
    This is an ongoing issue which hase not yet been resolved:
    Sentinel-2 downloader does not download images newer than August 28, 2024 #334.
    The current implication is that images newer than August 28, 2024 cannot be downloaded.

  • force-level1-csd: regardless of the previous bullet, support for S2C has been implemented.
    As GCS does not hold Landsat Collection 2 data, the option to download Landsat has been disabled.
    Use force-level1-landsat instead

  • force-level1-landsat has been updated to reflect changes in USGS' authentication.

• FORCE L2PS

  • Support for Sentinel-2C (as well as a future Sentinel-2D) was added.

  • Coregistration has been enabled for Landsat, too.
    A potential use case is to better align tier 2 data.

  • force-level2 was refactored. Little has changed for users. The logfiles are more verbose, and
    log messages generated by force-level2 and by the core process force-l2ps are more cleary separated.
    The queue file has got an additional status, wherein unsuccessfully processed images get FAIL status instead
    of DONE. The wrapper script forcel-l2ps_ was discontinued; force-level2 is now directly
    calling force-l2ps.

  • A new parameter STRICT_WATER_VAPOR was introduced, which will result in process abortion when the water
    vapor database does not hold a day-specific value, i.e. the usage of the climatology is forbidden.
    Applies to Landsat only.

  • A new parameter FILE_AOI was introduced.
    Users can now use an area of interest by specifying the path to a polygon-based vector file.
    This file will be used to mask the Level 2 products to the boundary of this AOI.
    The AOI will be reprojected (if necessary), and then rasterized to match the processed image.
    This operation is performed after the radiometric correction, but before the reprojection and tiling.
    It has little effect on computation time, but will considerably reduce output volume for images that are at the edge of your study area.

  • force-level2-report additionally outputs the compiled log data in a csv file for further inspection.

• FORCE HLPS

  • Support for Sentinel-2C (as well as a future Sentinel-2D) was added.

  • Analogous to Python UDFs, R UDFs can now be used through FORCE. This applies both to
    the TSA and UDF submodules. Three new parameters were added: FILE_RSTATS, RSTATS_TYPE,
    OUTPUT_RSP. At least two functions must be present in the UDF: force_rstats_init, as well as
    force_rstats_pixel or force_rstats_block (depending whether RSTATS_TYPE is set to PIXEL
    or BLOCK). To run PIXEL-type functions, you need to install the snow and snowfall packages.
    To run the FORCE components that are now compiled against the R API (mostly force-higher-level,
    but also force-qai-inflate), you need to provide two environment variables, i.e.
    R_HOME and LD_LIBRARY_PATH. You can do it like this:

    • export R_HOME=$(R RHOME)
    • export LD_LIBRARY_PATH=$R_HOME/lib
    • force-higher-level parameters.prm

    In the pre-built Docker container, everything is pre-configured already.

  • Added a new parameter: STREAMING = FALSE can disable the streaming strategy of FORCE HLPS.
    When TRUE (previous behaviour, and still the default), FORCE will perform reading, computing and
    writing at the same time. If FALSE, these operations are done after one another in sequential mode.
    Disabling streaming might be necessary for some UDFs that otherwise produce threading conflicts
    with the internally used OpenMP functionality. When using R UDFs, FORCE will issue a warning and
    gracefully disable streaming.

  • Added a new parameter: PRETTY_PROGRESS. HLPS will display progress information on screen.
    When TRUE (previous behaviour, and still the default), the progress information overwrites itself
    to produce a pretty displayal. However, this can cause error messages (or printing in UDFs) to be
    overwritten. If disabled (FALSE), the progress information will be simply be appended on screen (stdout).

  • Removed warning message that no output is produced when using the sampling submodule.
    It is now checked whether the files are actually written and will only warn if
    no input was detected or if no files were written.
    The behaviour for all other submodules stays the same.

  • It is now possible to specify the product types that should be used. This is to give the user more freedom
    with respect to data sources to be used.
    There are two new parameters:
    PRODUCT_TYPE_MAIN and PRODUCT_TYPE_QUALITY. The main product is usually a reflectance product like BOA.
    When using composites, you may use BAP. This can be anything, but make sure that the string can uniquely
    identify your product. As an example, do not use LEVEL2 as this will not filter products apropriately.
    Note that the product should contain the bands that are to be expected with the sensor used, e.g. 10 bands
    when sensor is SEN2A. The quality product type should be a bit flag product like QAI. When using composites,
    you may use INF. This can be anything, but make sure that the product contains quality bit flags as outputted
    by FORCE L2PS. As an exception, it is also possible to give NULL if you don't have any quality masks.
    In this case, FORCE will only be able to filter nodata values, but no other quality flags as defined with SCREEN_QAI.
    Feature requested by Marcel Schwieder and Felix Lobert.

  • In the TSA submodule, it is now possible to disable fitting a monotonic trend within the harmonic model.
    The new parameter HARMONIC_TREND = TRUE/FALSE was added. Default behaviour (as before) is TRUE.

  • After being deprecated for a while now, the SPLITS method has been removed from the code base (TSA submodule)

  • The Level 3 submodule now allows to use the internal scores for determining the best available observation as
    as a weight, i.e., not performing a selection, but a weighted average. This idea is taken from the MAJA WASP
    approach of Olivier Hagolle. A new parameter SELECT = TRUE/FALSE was introduced. with FALSE enabling
    the weighted average. Note that the INF and SCR products will not be produced when using this option.

  • The Level 3 submodule received the new parameters USE_CLOUDY and USE_HAZY, which can be used to allow
    observations with a cloud or haze score of less than 1% in the composite. On one hand, this reduces nodata gaps
    in the composite. On the other hand, it will include clouds or haze if there is no better observation.

  • The Level 3 submodule received a new parameter COMBINE_SCORES, which allows you to choose between additive or
    multiplicative combination of the individual scores into the total score. Additive combination is the previously
    used method and remains the default. Multiplicative combination can be used, which is much stricter in case one
    observation is characterized by a very poor suitability in a single score.

  • a new parameter OUTPUT_SUBFOLDERS was implemented that allows to output the data into subfolders under the tile
    folder instead of just dumping everything into the tile folder itself. It is planned that this feature will be
    expanded in the future. OUTPUT_SUBFOLDERS = FALSE remains the default.

  • UDF functions now receive sensor short names instead of the generic BLEND that was used before.
    This only applies if the non-interpolated time series is given to the UDF.
    If the interpolated time series is given to the UDF, BLEND is still passed into the function.

  • TSA outputs now carry the whole DATE_RANGE in their filename to reduce overwriting conflicts.

  • In the TSA submodule, INTERPOLATE = NONE has become the new default
    as it has become evident that the previous default INTERPLOATE = RBF confused users.

  • In the TSA submodule, the outlier/inlier detection has been switched off per default.

• FORCE AUX

  • new auxilliary program force-info, which replaces force effective immediately.
    The new program has the same functionality, but does not hardcode the available tools.
    Tools that are still missing the -i option are currently not listed though. This
    still needs to be done.

  • new auxilliary program force-init.
    This program will create a new project with reasonably named folders that
    can be used as a starting point for a new project with some suggestions
    on organizing things.
    Unused folders can be deleted and the naming and structure is a mere suggestion and by no
    means prescriptive or mandatory.
    This tool is especially meant for beginners.

  • new auxilliary program force-datacube-size.
    This program prints the size of your datacube, per sensor and in total.

  • new auxilliary program force-hist.
    This program computes the histogram of image values (can be vrt) and writes a csv table.
    This is intended to be used in a validation workflow.

  • new auxilliary program force-sample-size.
    This program computes the required sample size following Olofsson et al. 2013/2014,
    which should be used to properly validate a classification map.
    This is intended to be used in a validation workflow.

  • new auxilliary program force-stratified-sample.
    This program draws a stratified random sample based on a classification map and sample size computations.
    This is intended to be used in a validation workflow.

  • new auxilliary program force-map-accuracy.
    This program computes area-adjusted accuracies following Olofsson et al. 2013/2014.
    This is intended to be used in a validation workflow.

  • force-tabulate-grid has been updated to produce properly named output files.
    The default output file name is grid.kml, created in the current directory, using the KML format.
    -o can override the output file name (including path and extension).
    -f can override the file format, use the GDAL vector drive short name - thus grid can now be
    generated in any GDAL vector format. Both need to be given if not using defaults.

  • force-tile-extent has been updated to common Linux usage with options. Small changes for users are necessary,
    though all the functionality remained the same

  • Florian Katerndahl fixed the available resampling options in force-cube as a response to changes in GDAL.
    Thanks.

  • The CLI help of force-tile-finder has been corrected concerning the separator for the coordinates.

  • all aux scripts using parallel processing now use the number of available cores as default, not the number of input jobs.

• General

  • The Makefile as well as the codebase has been restructured.
    It is currently under active testing whether it simplifies or hinders development.
    Some changes may be reverted in the future.
    Usage is not affected.

  • Some changes in the Dockerfile have been made.
    Thanks to Peter A. Jonsson.

  • Unit test functionality was added, along with a small set of unit tests.
    This is far from complete at the moment.
    A new program force-unit-testing has been added.
    This is meant for DevOps pipelines only, not for regular users.

  • a pre-commit hook has been implemented, such that each commit in the develop branch is assigned an individual software version.
    This is a time-stamped version number.
    It should be communicated when reporting errors, bugs, etc.

This is a rather big collection of new featrues and many fixes.
I hope I didn't miss anything important in here.