Skip to content

Commit

Permalink
Compress normalized raster files
Browse files Browse the repository at this point in the history
  • Loading branch information
index-git committed Nov 1, 2021
1 parent 97fb045 commit c3616db
Show file tree
Hide file tree
Showing 5 changed files with 31 additions and 8 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#### Data migrations
### Changes
- [#169](https://github.com/LayerManager/layman/issues/169) [POST Workspace Layers](doc/rest.md#post-workspace-layers) accepts also compressed data files in ZIP format (`*.zip`) in `file` parameter. [PATCH Workspace Layer](doc/rest.md#patch-workspace-layer) accepts also data file in ZIP format (`*.zip`) in `file` parameter.
- [#503](https://github.com/LayerManager/layman/issues/503) Normalized GeoTIFF for raster files are also compressed.
- [#169](https://github.com/LayerManager/layman/issues/169) [GET Workspace Layer](doc/rest.md#get-workspace-layer) returns path to main file inside archive if zipped file was sent (key `file.path`).
- [#465](https://github.com/LayerManager/layman/issues/465) Fix situation, when Layman does not start if *.qgis file of the first layer with QML style does not exist. It was already fixed in v1.14.1.
- [#464](https://github.com/LayerManager/layman/issues/464) Fix publishing layers with unusual attribute names (e.g. `x,` or `Číslo`) and QML styles. It was already fixed in v1.14.1.
Expand Down
2 changes: 1 addition & 1 deletion doc/data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Subsequently, asynchronous tasks ensure following steps:
- data file chunks and completed data files are saved to [filesystem](#filesystem) (if sent [asynchronously](async-file-upload.md))
- vector data files are imported to [PostgreSQL](#postgresql)
- PostgreSQL table with vector data is registered to [GeoServer](#geoserver)
- raster files are normalized to GeoTIFF in EPSG:3857 with overviews (pyramids)
- raster files are normalized and compressed to GeoTIFF in EPSG:3857 with overviews (pyramids)
- normalized GeoTIFF is registered to [GeoServer](#geoserver)
- SLD file is saved to [GeoServer](#geoserver) and registered to WMS layer
- QGS file is created on [filesystem](#filesystem) and through QGIS server registered to [GeoServer](#geoserver)
Expand Down
2 changes: 1 addition & 1 deletion doc/rest.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ Processing chain consists of few steps:
- save file to workspace directory within Layman data directory
- save basic information (name, title, access_rights) into PostgreSQL
- for vector layers import the vector file to PostgreSQL database as new table into workspace schema, including geometry transformation to EPSG:3857
- for raster layers normalize raster file to GeoTIFF in EPSG:3857 with overviews (pyramids); NoData values are normalized as transparent only if Alpha band is not available and NoData is set for each band
- for raster layers normalize and compress raster file to GeoTIFF in EPSG:3857 with overviews (pyramids); NoData values are normalized as transparent only if Alpha band is not available and NoData is set for each band
- for vector layers publish the vector table as new layer (feature type) within appropriate WFS workspaces of GeoServer
- save bounding box into PostgreSQL
- for vector layers
Expand Down
21 changes: 16 additions & 5 deletions src/layman/layer/filesystem/gdal.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,12 +181,11 @@ def create_vrt_file_if_needed(filepath):
return vrt_file_path


def normalize_raster_file_async(workspace, layer, input_path, crs_id):
def normalize_raster_file_async(input_path, crs_id, output_file):
color_interp = get_color_interpretations(input_path)
result_path = get_normalized_raster_layer_main_filepath(workspace, layer)
bash_args = [
'gdalwarp',
'-of', 'GTiff',
'-of', 'VRT',
'-co', 'PROFILE=GeoTIFF',
'-co', 'INTERLEAVE=PIXEL',
'-co', 'TILED=YES',
Expand Down Expand Up @@ -242,9 +241,21 @@ def normalize_raster_file_async(workspace, layer, input_path, crs_id):
bash_args.extend([
'-t_srs', 'EPSG:3857',
input_path,
result_path,
output_file,
])
# print(' '.join(bash_args))
process = subprocess.Popen(bash_args, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return process


def compress_raster_file_async(workspace, layer, *, file_to_compress):
result_path = get_normalized_raster_layer_main_filepath(workspace, layer)
bash_args = [
'gdal_translate',
'-co', 'compress=lzw',
file_to_compress,
result_path,
]
process = subprocess.Popen(bash_args, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return process
Expand Down
13 changes: 12 additions & 1 deletion src/layman/layer/filesystem/tasks.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import os
import time
import tempfile

from celery.utils.log import get_task_logger

Expand Down Expand Up @@ -93,14 +94,24 @@ def finish_gdal_process(process):

input_path = layer_info['_file']['gdal_path']
vrt_file_path = gdal.create_vrt_file_if_needed(input_path)
process = gdal.normalize_raster_file_async(workspace, layername, vrt_file_path or input_path, crs_id)
tmp_vrt_file = tempfile.mkstemp(suffix='.vrt')[1]
process = gdal.normalize_raster_file_async(vrt_file_path or input_path, crs_id, output_file=tmp_vrt_file)
while process.poll() is None and not self.is_aborted():
pass
finish_gdal_process(process)

process = gdal.compress_raster_file_async(workspace, layername, file_to_compress=tmp_vrt_file)
while process.poll() is None and not self.is_aborted():
pass
if vrt_file_path:
try:
os.remove(vrt_file_path)
except OSError:
pass
try:
os.remove(tmp_vrt_file)
except OSError:
pass
finish_gdal_process(process)

process = gdal.add_overview_async(workspace, layername)
Expand Down

0 comments on commit c3616db

Please sign in to comment.