Skip to content

Commit

Permalink
Add 'file_limit' option to t.rast.series (OSGeo#2429)
Browse files Browse the repository at this point in the history
Add the option to set the file_limit (the maximum number of open files allowed for each r.series process), like the same option in t.series.aggregate.

Co-authored-by: Markus Neteler <neteler@osgeo.org>
  • Loading branch information
ecodiv and neteler committed Nov 7, 2023
1 parent 827de82 commit 28e6206
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 3 deletions.
12 changes: 12 additions & 0 deletions temporal/t.rast.series/t.rast.series.html
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,18 @@ <h2>DESCRIPTION</h2>
<em>t.rast.series</em> is a simple wrapper for the raster module
<b>r.series</b>. It supports a subset of the aggregation methods of
<b>r.series</b>.

<h2>NOTES</h2>

To avoid problems with too many open files, by default, the maximum
number of open files is set to 1000. If the number of input raster
files exceeds this number, the <b>-z</b> flag will be invoked. Because this
will slow down processing, the user can set a higher limit with the
<b>file_limit</b> parameter. Note that <b>file_limit</b> limit should not exceed the
user-specific limit on open files set by your operating system. See the
<a
href="https://grasswiki.osgeo.org/wiki/Large_raster_data_processing#Number_of_open_files_limitation">Wiki</a>
for more information.

<h2>Performance</h2>
To enable parallel processing, the user can specify the number of threads to be
Expand Down
16 changes: 13 additions & 3 deletions temporal/t.rast.series/t.rast.series.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,14 @@
# %option G_OPT_R_OUTPUTS
# %end

# %option
# % key: file_limit
# % type: integer
# % description: The maximum number of open files allowed for each r.series process
# % required: no
# % answer: 1000
# %end

# %flag
# % key: t
# % description: Do not assign the space time raster dataset start and end time to the output map
Expand All @@ -84,7 +92,6 @@
# % description: Propagate NULLs
# %end


import grass.script as grass
from grass.exceptions import CalledModuleError

Expand All @@ -104,6 +111,7 @@ def main():
memory = options["memory"]
nprocs = options["nprocs"]
where = options["where"]
max_files_open = int(options["file_limit"])
add_time = flags["t"]
nulls = flags["n"]

Expand Down Expand Up @@ -137,10 +145,12 @@ def main():
file.close()

flag = ""
if len(rows) > 1000:
if len(rows) > max_files_open:
grass.warning(
_(
"Processing over 1000 maps: activating -z flag of r.series which slows down processing"
"Processing over {} maps: activating -z flag of r.series which slows down processing.".format(
max_files_open
)
)
)
flag += "z"
Expand Down

0 comments on commit 28e6206

Please sign in to comment.