URL Transform callback as a Map option #5021

asheemmamoowala · 2017-07-21T21:30:30Z

We have quite a few tickets (#4740, #2918, #4917, and gl-native/#7455) related to modifying XHR requests made by mapbox-gl-js for various resources. While these tickets are each slightly different, they all include an element of inspecting and modifying the XHR request before it is sent.
The common use cases break down to:

sending authentication headers/cookies with cross-origin requests
adding custom headers to requests
altering URLs dynamically

This PR addresses all these concerns by providing a per-map option for a transformRequest callback.

Example usage

let Map = new Map({    
  style: 'mapbox://styles/mapbox/streets-v9',
  transformRequest: (url, resourceType) => { 
    if (url.startsWith('http://myHost') {
      return {
        url: url.replace('http:', 'https:'),
        headers: { 'my-custom-header': true},
        withCredentials: true
    }
});

benchmark	master `93a9922`	4740-per-map-transform 0933fb7
map-load	160 ms	103 ms
style-load	91 ms	80 ms
buffer	937 ms	905 ms
fps	60 fps	60 fps
frame-duration	6 ms, 0% > 16ms	6 ms, 0% > 16ms
query-point	0.95 ms	0.98 ms
query-box	52.74 ms	53.28 ms
geojson-setdata-small	5 ms	4 ms
geojson-setdata-large	134 ms	131 ms

Checklist

Allow transforming URLs before they are requested Resource transforms: alter URLs before requesting #4740
Allow adding custom headers (Similar to Allow user to set headers for XHR vector tile get #2918, partial of Allow custom done callback for RasterTileSource.loadTile #3123)
Allow sending request with cookie/credentials Allow Cross-site Source Data Request with Credentials #3874
document any changes to public APIs
post benchmark scores
manually test the debug page

// cc @anandthakker @jfirebaugh @kkaefer

jfirebaugh

Thanks @asheemmamoowala!

My main questions here are around the public API. See inline comments for specifics. Generally, I think this is a reasonable stopgap for custom sources, but we should replace it with custom sources when they're available, rather than maintaining two features with such substantial overlap.

jfirebaugh · 2017-07-26T00:18:24Z

bench/benchmarks/buffer.js

@@ -24,7 +24,7 @@ module.exports = function run() {
    const evented = new Evented();

    const stylesheetURL = `https://api.mapbox.com/styles/v1/mapbox/streets-v9?access_token=${accessToken}`;
-    ajax.getJSON(stylesheetURL, (err, stylesheet) => {
+    ajax.getJSON({ url: stylesheetURL}, (err, stylesheet) => {


Nit: either space after { and before }, or not -- not mixed (xN).

jfirebaugh · 2017-07-26T00:18:49Z

bench/benchmarks/buffer.js

@@ -82,14 +82,24 @@ module.exports = function run() {
    return evented;
 };

+class StubMap extends Evented {
+    constructor() {


Not necessary?

jfirebaugh · 2017-07-26T00:19:00Z

bench/benchmarks/buffer.js

+    }
+
+    _transformRequest(url) {
+        return { url: url };


return { url };

jfirebaugh · 2017-07-26T00:22:36Z

src/source/load_tilejson.js

-module.exports = function(options, callback) {
+module.exports = function(options, requestTransformFn, callback) {
+    if (!callback) {
+        callback = requestTransformFn;


Make requestTransformFn a required argument.

jfirebaugh · 2017-07-26T00:28:29Z

test/suite_implementation.js

@@ -126,7 +126,8 @@ function cached(data, callback) {
    });
 }

-sinon.stub(ajax, 'getJSON').callsFake((url, callback) => {
+sinon.stub(ajax, 'getJSON').callsFake((requestParameters, callback) => {


({url}, callback) => { (x3)

jfirebaugh · 2017-07-26T00:31:20Z

bench/benchmarks/buffer.js

@@ -107,7 +117,7 @@ function preloadAssets(stylesheet, callback) {
        }

        function getTile(url, callback) {
-            ajax.getArrayBuffer(url, (err, response) => {
+            ajax.getArrayBuffer({ url: url}, (err, response) => {


jfirebaugh · 2017-07-26T00:44:18Z

src/source/raster_tile_source.js

@@ -52,7 +52,7 @@ class RasterTileSource extends Evented implements Source {

    load() {
        this.fire('dataloading', {dataType: 'source'});
-        loadTileJSON(this._options, (err, tileJSON) => {
+        loadTileJSON(this._options, (url) => { return this.map._transformRequest(url, ajax.ResourceType.Source); }, (err, tileJSON) => {


loadTileJSON(this._options, this.map._transformRequest, (err, tileJSON) => {

Let loadTileJSON supply ajax.ResourceType.Source.

jfirebaugh · 2017-07-26T00:45:48Z

src/source/worker_source.js

@@ -7,7 +7,7 @@ import type {SerializedBucket} from '../data/bucket';
 import type {SerializedFeatureIndex} from '../data/feature_index';
 import type {SerializedCollisionTile} from '../symbol/collision_tile';
 import type {SerializedStructArray} from '../util/struct_array';
-


Nit: keep the whitespace between imports and exports.

jfirebaugh · 2017-07-26T00:50:05Z

src/source/geojson_source.js

 const EXTENT = require('../data/extent');
+const resolveURL = require('../util/mapbox').resolveURL;


Any particular reason for moving resolveURL? If it does move, it should go in something like util.js. mapbox.js is for Mapbox-specific URL munging.

@jfirebaugh util.js already depends on window, so putting resolveURL in util.js creates a cyclic dependency. resolveURL could be moved to ajax.js, or left in geojson_source.js which was its only usage anyways

jfirebaugh · 2017-07-26T00:56:05Z

src/ui/map.js

+     *  }
+     * });
+     */
+    setRequestTransform(transform: Function) {


Type transform more specifically than Function -- Function is like any: it turns off type checking.

Let's not make a setter for this, only a constructor option. This allows us to reduce internal coupling by passing a (normalized) transform function instead of the whole Map object. (If someone really needs dynamic behavior, they can put it inside the transform. But it's still a questionable idea: network requests may be triggered at any time, and we don't guarantee exactly when the transform gets called.)

Normalization looks like:

this._transformRequest = transform ? (url, type) => (transform(url, type || ajax.ResourceType.Unknown) || {url}) : (url) => ({url});

No need for _transformRequestCallback, and you can use map._transformRequest as a standalone function unconditionally from anywhere.

Also, is there a reason for the resource type to be optional?

Thanks for this suggestion. I didn't even consider the timing and order-of-execution issue with regards to Map#setRequestTransform.
reousrceType is optional for catching calls that don't provide a type.

Yermo · 2017-07-26T21:42:43Z

FWIW, the addition of this feature solves a big problem for me as I need to send an Authorization header with my requests. (I was on my way to implementing something like this, so thanks!)

jfirebaugh · 2017-07-27T17:51:05Z

src/util/ajax.js

+export type RequestParameters = {
+    url: string,
+    headers?: Object,
+    withCredentials? : Boolean


Let's make this a bit more extensible, and consistent with the Fetch API (which we may eventually switch to internally):

credentials?: 'same-origin' | 'include'

(It doesn't look like it's possible to emulate Fetch's 'omit' value using XHR, so I left that out.)

jfirebaugh · 2017-07-27T17:54:07Z

src/source/geojson_source.js

@@ -170,7 +170,7 @@ class GeoJSONSource extends Evented implements Source {
        const options = util.extend({}, this.workerOptions);
        const data = this._data;
        if (typeof data === 'string') {
-            options.url = resolveURL(data);
+            options.request = this.map._transformRequest(resolveURL(data));


Is this the only place where the type isn't provided? Native uses the Source type here; let's do the same and then make this parameter required.

jfirebaugh · 2017-07-27T17:54:46Z

src/style/image_sprite.js

@@ -31,24 +31,26 @@ class ImageSprite extends Evented {
    imgData: ?HTMLImageElement;
    width: ?number;

-    constructor(base: string, eventedParent?: Evented) {
+    constructor(base: string, eventedParent?: Evented, transformRequestCallback?: Function) {


Can the added parameter be required (in second position)?

… transform outgoing request.

asheemmamoowala · 2017-07-27T22:05:49Z

Closes #4740 and closes #3874

rayterrill · 2017-08-04T16:17:38Z

Any idea when this will be released? This looks like it'll fix some of the CloudFront Signed Cookies issues we're currently experiencing.

kkaefer · 2017-08-07T08:26:04Z

@rayterrill It'll be part of the next GL JS release. We're typically releasing new versions ~6 weeks. Mapbox GL JS v0.39 was released on July 21, so the next release will likely be end of August/beginning of September.

rayterrill · 2017-08-07T14:27:13Z

@kkaefer Thanks mate!

volkergraf · 2017-08-07T17:11:16Z

Just a Question .. the Transformation of the URL seems to work ok .. but now I'm getting a "Error: Unimplemented type: 3"

The Code looks like:

transformRequest: (url, resourceType) => {
              console.error(resourceType);
              alert("url: "+ url+"->"+ a_urls_to_blobs[url],);
              return {
                url: a_urls_to_blobs[url],
                headers: { "content-encoding" :"gzip"},
              }
            }

Do I have to set any type or something like that ?

asheemmamoowala · 2017-08-07T18:21:02Z

@volkergraf can you provide more context on the error you are seeing ? A reproducible case with jsfiddle would be helpful in debugging.

kkaefer · 2017-08-08T09:29:52Z

It sounds like the response is zipped but not decompressed correctly. Why are you setting "content-encoding": "gzip" for the request header? GET requests like this one don't have a body, so it doesn't make sense to specify a content encoding. Browsers automatically specify an Accept-Encoding that includes gzip, so there's no header that you need to set to benefit from transport compression.

volkergraf · 2017-08-08T10:08:35Z

@kkaefer Well this was only a test, trying to get rid of the "Error: Unimplemented type: 3" which did not work =)

volkergraf · 2017-08-08T10:24:43Z

@asheemmamoowala Ok .. well the Scenario is the following:

We are developing an iOS Application using Apache-Cordova.

WebKit has a Problem concerning AJAX-Calls using **file://..-**Urls .. but there is a workaround, using the cordova-file-plugin and the WXWebKitView (with a modified Initializer), allowing you to kinda use file://...-Urls to access the deployd files (that will reside in www/foo/bar), but you can NOT use the MapBox-GL.js source URL like "...www/foo/bar{z}/{x]/{y}.pbf, you have to transform it to a blob:file://var......11-22-33-66-767-88/1122.pbf blob-URL.

Therefore I created an associative-Array, containing the Original-URLs that the mapbox-gl.js-map requests and it's corresponding blob:-URLs .. a_urls_to_blobs[url]

I can definitively confirm that the mapbox-gl.js CAN load the Tiles trough the blob-URLs .. but it seems that using transformRequest: might require a few more Parameters to return .. to make the whole Thing run .. at the Moment I am returning simply the "blob:"-URLs .. perhapes I might have to add a type or something like that .. any Ideas ?

kkaefer · 2017-08-08T12:56:12Z

Error: Unimplemented type: 3 means that PBF decoding failed. It likely fails because the file wsa not decompressed correctly because the headers aren't set. Mapbox GL itself doesn't decompress anything; it relies on the browser infrastructure to do so.

rayterrill · 2017-10-20T21:47:39Z

I integrated this and implemented the transformRequest credentials functionality today using CloudFront signed cookies - Everything appears to be working great! Thanks all!

asheemmamoowala added GL native → GL JS For feature parity with Mapbox Maps SDK on a native platform under development labels Jul 21, 2017

asheemmamoowala changed the title ~~[wip] Initial implementation of URL Transform callback as a Map option~~ [wip] URL Transform callback as a Map option Jul 21, 2017

asheemmamoowala mentioned this pull request Jul 21, 2017

Resource transforms: alter URLs before requesting #4740

Closed

asheemmamoowala force-pushed the 4740-per-map-transform branch 3 times, most recently from c2d6446 to 5cf7788 Compare July 24, 2017 22:16

asheemmamoowala removed the under development label Jul 25, 2017

asheemmamoowala changed the title ~~[wip] URL Transform callback as a Map option~~ URL Transform callback as a Map option Jul 25, 2017

asheemmamoowala requested a review from jfirebaugh July 25, 2017 23:47

jfirebaugh suggested changes Jul 26, 2017

View reviewed changes

jfirebaugh approved these changes Jul 27, 2017

View reviewed changes

Implement Map.options.transformRequest - a callback function that can…

e26c9bc

… transform outgoing request.

asheemmamoowala force-pushed the 4740-per-map-transform branch from 293f965 to e26c9bc Compare July 27, 2017 21:42

asheemmamoowala merged commit b528524 into master Jul 27, 2017

asheemmamoowala deleted the 4740-per-map-transform branch July 27, 2017 22:10

This was referenced Aug 4, 2017

Allow Cross-site Source Data Request with Credentials #3874

Closed

Allowing dependency injection for global classes like XMLHttpRequest #5104

Closed

jmunowitch mentioned this pull request Sep 14, 2017

Upgrade to Mapbox-gl-js 0.40.0 archesproject/arches#2410

Closed

benrudolph mentioned this pull request Mar 9, 2018

Add transformRequest to ReactMap factory alex3165/react-mapbox-gl#554

Closed

datwelk mentioned this pull request Jun 6, 2018

[ios] Provide a custom NSURLSessionDelegate? mapbox/mapbox-gl-native#11888

Closed

ac87 mentioned this pull request Feb 14, 2021

URL Request Transform mapbox/mapbox-maps-android#106

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

URL Transform callback as a Map option #5021

URL Transform callback as a Map option #5021

asheemmamoowala commented Jul 21, 2017 •

edited

Loading

jfirebaugh left a comment

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

jfirebaugh Jul 26, 2017

asheemmamoowala Jul 26, 2017

jfirebaugh Jul 26, 2017

asheemmamoowala Jul 26, 2017

Yermo commented Jul 26, 2017

jfirebaugh Jul 27, 2017

jfirebaugh Jul 27, 2017

jfirebaugh Jul 27, 2017

asheemmamoowala commented Jul 27, 2017 •

edited

Loading

rayterrill commented Aug 4, 2017

kkaefer commented Aug 7, 2017

rayterrill commented Aug 7, 2017

volkergraf commented Aug 7, 2017

asheemmamoowala commented Aug 7, 2017

kkaefer commented Aug 8, 2017

volkergraf commented Aug 8, 2017

volkergraf commented Aug 8, 2017

kkaefer commented Aug 8, 2017

rayterrill commented Oct 20, 2017

		const EXTENT = require('../data/extent');
		const resolveURL = require('../util/mapbox').resolveURL;

URL Transform callback as a Map option #5021

URL Transform callback as a Map option #5021

Conversation

asheemmamoowala commented Jul 21, 2017 • edited Loading

Example usage

Checklist

jfirebaugh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yermo commented Jul 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asheemmamoowala commented Jul 27, 2017 • edited Loading

rayterrill commented Aug 4, 2017

kkaefer commented Aug 7, 2017

rayterrill commented Aug 7, 2017

volkergraf commented Aug 7, 2017

asheemmamoowala commented Aug 7, 2017

kkaefer commented Aug 8, 2017

volkergraf commented Aug 8, 2017

volkergraf commented Aug 8, 2017

kkaefer commented Aug 8, 2017

rayterrill commented Oct 20, 2017

asheemmamoowala commented Jul 21, 2017 •

edited

Loading

asheemmamoowala commented Jul 27, 2017 •

edited

Loading