Rendering raster tiles with WebGL #12008
Conversation
|
Impressive work @tschaub! And it fits very nicely into the existing tile/source/layer structure. Unfortunately I do not have time to give this an in-depth review at the moment, but from a quick glance I'm confident you're on the right path. |
|
This is impressive indeed. I hope @tschaub won't mind me skipping the pleasantries and going straight to comments. Regarding high-level architecture, The whole idea of doing fragment shaders in WebGL is combining disparate raster sources; but one of the difficulties I faced in #11810 was how to ensure that several tile sources shared the same tile grid (granted, this is legacy from the Leaflet TileLayer.GL naïve implementation). Now, OpenLayers has So, I'm starting to think that (ideally) tile sources should be projected and stitched into viewport-sized canvases, then do the same for non-tiled raster sources as well, and then all those data canvases should be passed to a shader to calculate the final visible colour. I'll try to explain with some crappy ASCII-art diagram: I think this idea breaks several paradigms, but it's been slowly coalescing into being "the right thing™" in my head. And yes working with invisible non-RGBA framebuffers is a mess, but I think this is the right pipeline in the long run. Back to actual stuff:
16-bit GeoTIFFs/textures and There are basically two ways to go: assume the lowest capabilities (WebGL1, no (as a side note, Choosing between WebGL1 and WebGL2 capabilities also impacts the availability of 3D textures (not to be confused with cubemapped textures) - with 3D textures, a GeoTIFF with an arbitrary number of samples (in 8-, 16- or 32-bit format) can be loaded into one single texture; without 3D textures, onlt 32 bits can be loaded into a texture. This forces sample (AKA channel AKA band) selection and possibly byte-packing. I have a few Sentinel-2 GeoTIFFs with 5+ samples, each of them 16-bit (courtesy @EOX-A / @Schpidi ). GeoTIFFs with 5+ samples are an important use case. Re
That is simply awesome. 😎 Re
I am against this approach - I just don't think that forcing Polish notation (or encouraging it) is good. The current approach is This highlights one of my intentions with exposing GLSL (as strings passed to constructors): users should be able to learn just one shading language; and I think that the best shading language to be learn is GLSL (because of existing learning resources on complex shaders); not ArcGIS' raster algebra language, not QGIS's, not SentinelHub's, and definitely not an AST. I think that writing partial GLSL shader code (i.e. the main function) and providing aux shader functions (e.g. texture byte unpacking, colour ramps, etc) is OK. Yes, this raises the question of what is the name of the output fragment variable (which changes between WebGL1 and WebGL2) and whether the I think this is all the WebGL-related thinking my brain can do for now. That was quite the braindump, though 😄 |
There was a problem hiding this comment.
I haven't yet taken the time to dive deep into the tile rendering logic, but the general layout of the new classes and their interactions looks great.
About the GeoTIFF source, a naive question: would this source work if used with a regular Tile Image layer?
I think it's great to be able to use a styling language similar to the webgl vector layers, even though the context is completely different. Sure, using polish notation (inspired by the MapBox style spec btw) may look weird at first but it builds upon a coherent JSON-based styling API, so IMO it's the right direction.
Thanks, @tschaub
@IvanSanchez - If you squint at it the right way, your suggestion shares ideas with an older rendering architecture where a single map renderer was responsible for doing more composition work. The current design treats layers (mostly) independently when rendering. It may be that at some point we return to a single map renderer responsible for composition, but we intentionally changed things to allow layers to work more independently so we could reintroduce WebGL rendering in an incremental way. I think at this point it is more important to discuss the layer and source APIs - knowing that the specifics of the implementation will change over time. In this branch, I've proposed new types of Tile, Source, and Layer. While the API for those new types is more constrained that what you have proposed (for example, the user only controls the layer |
@IvanSanchez - If you could describe the specifics of this use case, that would be helpful. (You may have already put together an example of this that I have missed.) |
| }, ${output2}, pow(clamp((${input} - ${stop1}) / (${stop2} - ${stop1}), 0.0, 1.0), ${numberToGlsl( | ||
| interpolation | ||
| )}))`; | ||
| result = `mix(${output1}, ${output2}, pow(clamp((${input} - ${stop1}) / (${stop2} - ${stop1}), 0.0, 1.0), ${exponent}))`; |
There was a problem hiding this comment.
Ideally we would not have to repeat the input expression for each stop. For example, in the cog-math.js example, the interpolate expression serializes to this:
mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(mix(vec4(0.7490196078431373, 0.7490196078431373, 0.7490196078431373, 1.0), vec4(0.8588235294117647, 0.8588235294117647, 0.8588235294117647, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - -0.2) / (-0.1 - -0.2), 0.0, 1.0), 1.0)), vec4(1.0, 1.0, 0.8784313725490196, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - -0.1) / (0.0 - -0.1), 0.0, 1.0), 1.0)), vec4(1.0, 0.9803921568627451, 0.8, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.0) / (0.025 - 0.0), 0.0, 1.0), 1.0)), vec4(0.9294117647058824, 0.9098039215686274, 0.7098039215686275, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.025) / (0.05 - 0.025), 0.0, 1.0), 1.0)), vec4(0.8705882352941177, 0.8509803921568627, 0.611764705882353, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.05) / (0.075 - 0.05), 0.0, 1.0), 1.0)), vec4(0.8, 0.7803921568627451, 0.5098039215686274, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.075) / (0.1 - 0.075), 0.0, 1.0), 1.0)), vec4(0.7411764705882353, 0.7215686274509804, 0.4196078431372549, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.1) / (0.125 - 0.1), 0.0, 1.0), 1.0)), vec4(0.6901960784313725, 0.7607843137254902, 0.3803921568627451, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.125) / (0.15 - 0.125), 0.0, 1.0), 1.0)), vec4(0.6392156862745098, 0.8, 0.34901960784313724, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.15) / (0.175 - 0.15), 0.0, 1.0), 1.0)), vec4(0.5686274509803921, 0.7490196078431373, 0.3215686274509804, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.175) / (0.2 - 0.175), 0.0, 1.0), 1.0)), vec4(0.5019607843137255, 0.7019607843137254, 0.2784313725490196, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.2) / (0.25 - 0.2), 0.0, 1.0), 1.0)), vec4(0.4392156862745098, 0.6392156862745098, 0.25098039215686274, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.25) / (0.3 - 0.25), 0.0, 1.0), 1.0)), vec4(0.3803921568627451, 0.5882352941176471, 0.21176470588235294, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.3) / (0.35 - 0.3), 0.0, 1.0), 1.0)), vec4(0.30980392156862746, 0.5411764705882353, 0.1803921568627451, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.35) / (0.4 - 0.35), 0.0, 1.0), 1.0)), vec4(0.25098039215686274, 0.49019607843137253, 0.1411764705882353, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.4) / (0.45 - 0.4), 0.0, 1.0), 1.0)), vec4(0.18823529411764706, 0.43137254901960786, 0.10980392156862745, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.45) / (0.5 - 0.45), 0.0, 1.0), 1.0)), vec4(0.12941176470588237, 0.3803921568627451, 0.07058823529411765, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.5) / (0.55 - 0.5), 0.0, 1.0), 1.0)), vec4(0.058823529411764705, 0.32941176470588235, 0.0392156862745098, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.55) / (0.6 - 0.55), 0.0, 1.0), 1.0)), vec4(0.0, 0.27058823529411763, 0.0, 1.0), pow(clamp((((color.g - color.r) / (color.g + color.r)) - 0.6) / (0.65 - 0.6), 0.0, 1.0), 1.0))
We could factor out this:
float interpolateInput = (color.g - color.r) / (color.g + color.r);
And use interpolateInput in the expression instead. We could make this work by having the toGlsl function append additional statements to the context that would be included when composing the shader. Or we could have toGlsl return more than just a string expression.
I'm not sure how to measure whether having the fragment shader evaluate this (color.g - color.r) / (color.g + color.r) 20 extra times per pixel has a significant impact. Curious if others think this type of thing is worth optimizing.
There was a problem hiding this comment.
This was also something I was wondering about. AFAIK the GPUs do some optimizations when compiling shaders but this is hardware dependant I think? Also some operations are more costly than others. Here though, I don't know if the extra computations have any impact at all. Probably @IvanSanchez would know more about this?
|
First off, this work is greatly appreciated and something I personally waited years for. We did a similar thing with COG-Explorer but had to use shims and canvas rendering techniques which were not really portable and quite hacky. Regarding the use of colorscales: In my opinion the best way is to use a texture for the coloring part. We did this in the plotty.js library. Of course, this involves a little more extra work, setting up the textures and such but then the resulting shader is quite clean and you get stuff like interpolation for free. Regarding the used polish notation expression language: to be honest, I'm not too excited about that but I understand the need for an intermediate language/AST to later serialize to the shading language. I think using an expression language like we did in plotty.js is quite easy and understandable for users. But this can be simply be used beforehand to then be transformed to the polish notation. On the other hand, having the possibility to directly pass shader code is a very important feature for when you really want to fine-tune the used code and mix it with other tools. I think this is key to have. For my uses-cases the biggest drawback right now is the restriction to 3/4 band Uint8 data. I'd like to be able to dynamically visualize Landsat 8 or Sentinel-2 images (>10 Bands, Int 16) with image masks and such and mix them using algorithms like NDVI with a dynamic colorscale. As demonstrated with the COG-Explorer this is already possible, although not necessarily straightforward. I'd very much like to help moving this topic forward, if you are interested. Regarding the current GeoTIFF reading: I see that the current implementation treats the files as RGB(A) but uses the raw I think the proposed PR should work well if we have an external grid of GeoTIFFs, which is also a requirement for us. Please let me know what you think. |
|
Thanks for the feedback, @constantinius. A few comments inline below:
I can imagine a way to extend the current
Could the data for the mask be derived from the same GeoTIFF, or are you talking about masking one source of data using a separate source? If you have any example data sets, it would be great to agree on an example use case that we want to support and then develop the required functionality.
I do think we can make it possible to allow people to write their own shaders. Though for the reasons I gave above, I don't think it is wise to have the Layer API be primarily oriented around composing user-provided shader snippets. We don't need to ensure that we have supported all the possible use cases before getting something in. But we do want to ensure that we don't provide an API that makes it impossible for the implementation to evolve without breaking existing use cases. |
@constantinius - The latest commit now allows data tiles to have an arbitrary number of bands. This means you can configure a GeoTIFF source using a single GeoTIFF with > 4 bands or many GeoTIFFs with one band each. Note that the GeoTIFF source is not restricted to reading Uint8 data. You can use Int16, Int32, Float32, etc. sources as well. It is true that internally this data is quantized to 256 values. This is an implementation detail that we can change - and as with the initial limit on the number of bands, it is an implementation detail that we can change without breaking the Layer/Source/Tile API (I was able to remove the limit on the number of bands without changing the API or examples). If we can focus on the functionality and the API that we want to provide instead of the specifics of the implementation, I think it will be easier to arrive at something that makes sense in the library. If you can provide details about the example functionality that you'd like to see - ideally with publicly available data - that would be great. It is also worth pointing out that while it is now possible to configure data tile sources (like the GeoTIFF source) with an arbitrary number of bands, this is not necessarily the best application design. For example, if you configure a GeoTIFF source with 10 single-band GeoTIFFs, that means 10 source tiles need to be loaded before we can start rendering. This is true even if in the end you are only using two or three bands in a band math expression. If I were building an application that allowed users to dynamically explore different metrics using different bands, I would dynamically generate new sources with just the required bands. It will lead to an improved user experience to only load the bands required for the currently selected metric rather than to preemptively load all bands for all of the possible metrics. |
That depends on the dataset in question. For Landsat-8 everything is in its own file, also the data masks. See the example scene here.
That's great news! Thanks for taking care of that.
Sorry, I agree, getting I tend to get carried away with details. I would love to be able to do what is currently possible on COG-Explorer with Landsat data (e.g. here, please note the visualization options behind the top right wrench icon). We provide a couple sample images/tilesets for non-commercial use (CC-BY-NC-SA). From this Uint16 data we would love dynamically compute true color, false color, and NDVI representations, if possible using dynamic color scales.
I totally agree, that pre-loading bands is definitely not a good idea! I was going from the implementation of the COG-Explorer which looked at the bands used in the expression and only loaded and cached those. Now when the user changed the visualization and thus the used bands, only new data would be fetched whereas already fetched data would be re-used, which is in my opinion the key point: we actually want to reduce the amount of data transmitted even when users change the visualization and fiddle with the expressions, variables, etc. This is why I think that re-creating new sources (and thus new geotiff.js instances) will not be sufficient, as we would loose the already downloaded data. Or am I wrong in this assumption?
I agree with that. I'm really fond of the shaping architecture and the "OpenLayerness" of the implementation. I also agree that some details can be delayed so that an initial implementation can be shipped and not everything has to be there from the start. I just want to note that there are already quite some investments in both the COG-Explorer and the GlTiles implementation which I would hope to not loose. I hope that is understandable. |
|
Hi, and thank you for the amazing work on this so far! Sorry for the newbie question but how can I test it? I tried |
|
Rebased to resolve conflicts in |
|
@howff You'll have to clone the OpenLayers repository locally: Now you can play with the examples locally. Now you have a linkable package. To test it in your project, go into the project and Now you can play with your locally linked version of the package. |
Well, basically the current implementation is agnostic to what the bands contain by the time they are read in with function(geoTiffImage, pixelBounds) {
return geoTiffImage.readRGB({window: pixelBounds});
}@tschaub, do you think this would expose too many internals to the API? |
We don't currently set up a worker for decoding, but my intention was to add that. The function (image, options) {
return image.readRGB(options);
}That would allow people to modify options before passing them along. The other issue this brings up is interleaving. We currently rely on an array of typed arrays (one per band). The An alternative would be for us to always call |
|
Dear all, any news about WebGL tile layer to the final version? When do you plan to include this in the final version? |
|
@mirosvanek The goal is to bring this into a release before the FOSS4G conference, i.e. last week of September. |
There was a problem hiding this comment.
I think this is ready to be merged. I finished the open tasks that @tschaub had identified. @tschaub's work - especially the API - proved to be a solid foundation. In addition to the mentioned tasks and a couple of new examples I made a few minor fixes, like passing the transition option from the source to the renderer, and fixing alpha handling during tile transitions.
Thanks @tschaub for this effort! Please merge when you think that my commits look good too, or suggest changes otherwise.
|
Thanks for this valuable contribution! It's impressive work and this will be immediately useful for people that wait to adopt COGs with OpenLayers. I'm currently experimenting with this branch and I hope it is okay to add some questions that I could not resolve from reading and experimenting with this PR? Just as an experiment I'm trying to just make a page where a user can specify the link of a random COG and then visualize it.
These questions came up for me because 1+2 work automatically in the Leaflet plugin that can render COGs. And it seems the COG example in this PR points to the file has all the necessary metadata, so why not use it? The GeoTiff.js source seems to include some code for auto-detection, but I could not make it work in the COG example. If anything of this would work, it would probably be a good example, too. Any feedback is highly appreciated. Disclaimer: I'm not very experienced with OpenLayers nor with COGs, so I hope there's nothing too obvious that I'm missing here. |
|
Thanks for the feedback, @m-mohr.
I've updated the GeoTIFF source so it now reads no-data values from the metadata when possible. The I'll add an additional comment on the other two points a bit later. |
The GeoTIFF source does currently read the extent and projection related metadata from the imagery. The problem is that these properties would be most useful to have when creating the view for a map, and there isn't currently a good way to have the source provide that information to the map or the app developer. I've opened #12644 to discuss ideas around auto-generating a view based on source (or layer) metadata, and we can collect additional ideas on the subject there. |
|
Thanks, @ahocevar for the great collaboration on this. And thanks to others for the input and suggestions. |
|
Thanks for your thoughts on my issues. I agree on the proposal in #12644 and thanks for the improvement on no-data values. |
|
I'm still a little bit sad that #11810 didn't make it, but it's good to see this functionality finally merged in. |
This pull request adds support for rendering raster tiles with WebGL. It builds on the existing WebGL utilities being developed for vector data rendering and follows existing patterns for tile rendering. The work is inspired by the proposed contribution in #11810, but takes a different approach.
I've put this together in a single branch to demonstrate the direction and end goal, but will be pulling apart pieces of this functionality as separate pull requests for easier review.
New modules
ol/DataTile.jsData tiles are a new class of tile that are meant to allow working with arbitrary
TypedArraydata. Currently,Uint8ArrayandUint8ClampedArrayare supported, but additional types could be added. TheDataTileclass is useful for applications that render non-image raster data – this could be data that is generated at runtime or data that is loaded from something like a GeoTIFF. The constructor is meant to provide flexibility in loading data – taking aloaderfunction that returns a promise for data – and we should not need to create additional subclasses.ol/source/DataTile.jsThe data tile source works with the new data tiles described above. The source allows configuration of the tile grid and provides a
getTile()method for consumers (i.e. renderers).ol/layer/WebGLTile.jsThe WebGL tile layer creates a vertex shader and a fragment shader for rendering raster tiles with WebGL. The shaders are generated by parsing a
styleexpression that is passed to the layer – building on the style parsing used by the WebGL points layer. The user-provided style generates shader code that does two things: 1) optionally transforms tile data with acolorexpression and 2) applies a pipeline of operations that transform the color withbrightness,contrast,saturation,exposure, orgammaexpressions.We can continue to build on the style syntax to provide additional flexibility in generating shaders. For example, we can extend the current band math expressions to allow access to neighboring pixels. If we want to, we could also make it convenient for people to provide hand-written shaders of their own (this is already possible – but maybe not convenient – by creating a custom layer type and using the new renderer described below).
ol/renderer/webgl/TileLayer.jsThe WebGL tile layer renderer works with the layers described above, compiling and executing shader programs as part of map rendering. The renderer logic should be familiar from the existing Canvas tile layer renderer. One notable difference is that the new renderer maintains its own cache of tile textures (see below). With the existing sources and renderers, the source maintains a tile cache that is managed by the renderer. This is awkward because the renderer is responsible for things like pruning from the source's cache and the source is responsible for things like knowing that it needs to cache tiles in multiple projections for multiple consumers. It would be more straightforward if tile sources were only responsible for generating tiles and consumers could decide what to cache.
ol/webgl/TileTexture.jsThis provides a single interface wrapping a tile (image tile, data tile, etc.) and a texture. The tile texture monitors the tile state and uploads texture data to the GPU on tile load.
ol/source/GeoTIFF.jsWhile GeoTIFF loading and rendering is not really WebGL specific, I've added this functionality here based on inspiration from #11810. The GeoTIFF source extends the data tile source described above. The loading and parsing of GeoTIFF data is handled by
geotiff.js, which is added as a dependency of this source. The source provides flexibility in working with different configurations of underlying "source" data.nodatavalues as transparent.bandexpressions to do band math before rendering a final color. Note that tiles for each GeoTIFF source will be loaded even if all bands are not used in an expression (so you can improve the user experience by configuring only the required bands as sources).The source’s tile grid is configured based on the tile layout and overviews within the provided GeoTIFFs. For GeoTIFFs with external overviews, it is possible to provide a list of URLs as
overviews.Each of the GeoTIFF sources can also be configured with
minandmaxvalues used to scale values between 0 and 1 (the defaults are derived from the bit depth in the GeoTIFF, but it is common to find data with custom gain/bias or data ranges). This part may change, but I like the idea that layers assume "normalized" data values.Examples
Basic "hello world" WebGL tile rendering
webgl-tiles.html
Nothing too fancy here. Demonstrates tile transitions and rendering of alt-z tiles with an OSM source.
WebGL tile layer style
webgl-tile-style.html
Adjusting
stylevariables based on application state to adjust things like exposure, contrast, and saturation.Data tiles
data-tiles.html
Data tiles are a building block. This example renders tile ids in the tile data, but any arbitrary data can be rendered.
Cloud Optimized GeoTIFF (COG)
cog.html
This renders a 3-band publicly hosted Sentinel 2 COG.
GeoTIFF with external overviews
cog-overviews.html
Some GeoTIFFs are hosted with external overviews. This example loads data of this type, rendering a false color composite using bands from three different sources.
Normalized Difference Vegetation Index (NDVI)
cog-math.html
The
styleproperty of a WebGL tile layer accepts acolorproperty that can perform math expression onbandvalues. In this case, NDVI is calculated from red and near infrared bands. The NDVI values are then mapped to colors with aninterpolateexpression.NDVI+NDWI from two 16-bit COGs
cog-math-multisource.html
Similar to the NDVI example, but the source data comes from two 16-bit GeoTIFFs with different resolutions, and the result of the NDVI and NDWI calculations is used directly as RGB color.
GeoTIFF tile pyramid
cog-pyramid.html
The GeoTIFF in this example is organized as tile pyramid, described with metadata following the STAC Tiled Assets
extension. A separate layer is created for each tile in the pyramid. The lowest resolution layer is used as placeholder while higher resolutions are loading.
Sea level (WebGL version)
webgl-sea-level.html
The example uses a
colorexpression to convert normalized pixel values to elevation measures. Then elevation measures are then mapped to color values with aninterpolateexpression. Thevaroperator is used in the colors array to choose the stop value based on thelevelstyle variable. As you drag the slider,layer.updateStyleVariables()is called to update the variables used in the style.Shaded relief (WebGL version)
webgl-shaded-relief.html
The example uses
bandexpressions with pixel offset to sample neighboring pixels for calculating slope and aspect. Like in the Sea level example, user input is passed to the style as style variables.Things I've decided not to do (for now):
OES_texture_floatto work with Float32Array, Int16Array, or Uint16Array data tiles (the latter are scaled as Uint8Array values).Things that still need done: