Refactor Source interface to allow/prepare for extensibility #2667

anandthakker · 2016-06-03T21:32:57Z

Replaces #2648

Inverts TilePyramid and Source dependencies. TilePyramid now essentially serves as a common wrapper class for dealing with sources.
Renames TilePyramid to SourceCache. This is partly aspirational, and partly to make it more reasonable to have things called source that are actually TilePyramids 😁
Eliminates the ability to create a Source independently of a Map instance - i.e., no more new mapboxgl.GeoJSONSource(); sources must be created with map.addSource(id, json).
Eliminates bucketStats, as it is no longer needed (so @lucaswoj assures me)
Source types are now classes with constructor signature (id, options, dispatcher) => Source and an optional workerSourceURL static property
The Source provided by the factory function is an object implementing:

id: string
tileSize: number
minzoom: number
maxzoom: number
roundZoom: boolean
reparseOverscaled: boolean
isTileClipped: boolean

loadTile: (tile: Tile, cb: (err, ?data) => void) => void
abortTile: (tile: Tile) => void
unloadTile: (tile: Tile) => void
serialize: () => Object // plain JS
?prepare: () => void

// @fires `load` to indicate source data has been loaded, so that it's okay to call `loadTile`
// @fires `change` to indicate source data has changed, so that any current caches should be flushed

WorkerSourceURL points to a script, which, when executed in the worker thread, must call self.registerWorkerSource(WorkerSource), where WorkerSource is a class that the worker will construct with new WorkerSource(actor, styleLayers).

WorkerSource instances may implement a loadTile method; if they do, then this will be used by the worker when its 'load tile' target is invoked with a type: 'source-type' parameter (e.g. type: 'geojson'). Any other methods on workerSource instance can be targeted by the main-thread Source via dispatcher.send('source-type.methodname', params, callback).

Questions:

Does the above design have broad (enough) approval?
The Tile objects are used by various source types to maintain tile-specific state; ideally, this implicit contract between TilePyramid and Source should be made explicit in the Source interface... but I think this is perhaps out of scope for this particular PR - perhaps this would be naturally addressed in Implement cross-source symbol placement #2703, or else could be ticketed separately?
I'm not sure what the data yielded by loadTile should look like. Currently, the only thing I can see being needed by code outside of the particular source is bucketStats, which the pyramid uses to emit the tile.stats event. @lucaswoj can you weigh on in this?

Remaining Tasks:

jfirebaugh · 2016-06-03T21:47:59Z

js/style/style.js

@@ -66,12 +72,36 @@ function Style(stylesheet, animationLoop) {
        this.glyphSource = new GlyphSource(stylesheet.glyphs);
        this._resolve();
        this.fire('load');
+
+        if (validateStyle.emitErrors(this, validateStyle(stylesheet))) return;


Why did this move? Don't we want to continue to short circuit if there's a validation error?

Hm, yup - not quite sure when/how this happened, but it's a mistake. Thanks!

jfirebaugh · 2016-06-03T22:21:14Z

Regarding Source.is, addSource, and createSource: I think we could take either of the following approaches:

Remove the ability to create or add a source other than addSource(id, json), where json is what you'd see in the style spec. (No more new mbgl.GeoJSONSource(), etc.)
Not go the factory function route just yet. Do whatever additional configuration is needed (e.g. setting dispatcher) on the source only when addSource is called.

jfirebaugh · 2016-06-03T22:25:53Z

It's weird to rely on sources presenting an id property with the correct value.

Seems fine to me. The reason that id is a separate parameter in addSource, and not included in the object like it is with layers, is that the style spec uses an id-keyed object rather than an array. This is probably a design mistake and we should been consistent: use arrays, validate id uniqueness externally. Anyway, I think of id as being an intrinsic property for both layers and sources.

jfirebaugh · 2016-06-03T22:35:41Z

js/style/style.js

@@ -68,10 +76,32 @@ function Style(stylesheet, animationLoop) {
        this.fire('load');
    }.bind(this);

-    if (typeof stylesheet === 'string') {
-        ajax.getJSON(normalizeURL(stylesheet), loaded);
+    var sourceTypesLoaded = function(err) {


Can you explain this part of the code? It looks like it's loading support for core source types asynchronously? That seems odd to me, both that it should be async, and that something like addSourceType should be the responsibility of Style and attached as a non-underscored method.

Would this comment tie in here?

@jfirebaugh Yeah, once this refactor settles, I think this async weirdness can be eliminated. It's here now only as a quick and dirty way to validate/show that addSourceType works, but ultimately there's no reason that core source types need to be added asynchronously through the addSourceType interface.

I do think addSourceType should:

(a) be a public, non-underscored method -- because its purpose is to allow third-party source types can be plugged in, which is the motivating reason for this PR; and

(b) be async -- because if a source type needs to provide a WorkerSource implementation, then the process of having all the workers load that implementation via importScript is async.

mourner · 2016-06-07T13:48:07Z

I'm still not sure if we want to have geojson-clustered as a separate type for clustering, and generally have different type names for the same source with slightly different interpretation. I'm more comfortable with the current API because the source is GeoJSON whether clustered or not — clustering is a matter of interpretation. Suppose we want to make a plugin for clustered TopoJSON — would we have a separate source named topojson-clustered and somehow reuse parts of both geojson-clustered and topojson sources?

anandthakker · 2016-06-07T14:15:07Z

@mourner I definitely see what you're saying, but I'm not 100% sold on the idea that different source types should only map to the format of the original source data being used. I agree that clustering is an interpretation of the data; but then would hex-bins and square-bins also just be interpretations that belong as options on GeoJSONSource?

anandthakker · 2016-06-07T15:23:30Z

js/source/source.js

+
+    if (source.id !== id) {
+        throw new Error('Expected Source id to be ' + id + ' instead of ' + source.id);
+    }


Actually, I think it probably just makes more sense to do source.id = id here.

anandthakker · 2016-06-09T14:44:53Z

@lucaswoj @jfirebaugh @mourner Updated the top of the ticket with more complete description of the overall design and capturing outstanding questions/tasks from the reviews thus far (so we should now be okay to rebase w/o losing key discussion points).

anandthakker · 2016-06-09T14:45:07Z

Transcribing from a diff comment:

@jfirebaugh : [It] seems odd to me, both that it should be async, and that something like addSourceType should be the responsibility of Style and attached as a non-underscored method.

@anandthakker : I do think addSourceType should:

(a) be a public, non-underscored method -- because its purpose is to allow third-party source types can be plugged in, which is the motivating reason for this PR; and
(b) be async -- because if a source type needs to provide a WorkerSource implementation, then the process of having all the workers load that implementation via importScript is async.

We should validate the custom source architecture and build some demo sources before documenting the internals publicly.

anandthakker · 2016-07-20T22:02:10Z

🎉

mourner · 2016-07-22T11:41:03Z

@lucaswoj For some reason, the build fails in master since the merge: https://circleci.com/gh/mapbox/mapbox-gl-js/4209

lucaswoj · 2016-07-22T11:54:38Z

Thanks for the heads-up @mourner. I'll try to resolve ASAP.

hdrdavies · 2016-08-17T16:48:08Z

@mourner how can one now configure clustering as was possible before i.e.

        let sourceObj = {
            type: 'foo',
            cluster: true,
            clusterRadius: 50,
            clusterMaxZoom: 14,
            data
        };

        map.addSource('foo', sourceObj);

mourner · 2016-08-17T16:54:25Z

@hdrdavies same as before, as you can see on the example https://www.mapbox.com/mapbox-gl-js/example/cluster/

hdrdavies · 2016-08-18T14:30:58Z

@mourner thanks very much!! It seems I'm getting other strange constructor errors even though I'm not using new GeoJsonSource anymore...

Do you think you guys will be adding support for the clustering of custom HTML markers made using new mapboxgl.Marker ? Maybe I'm missing something in the documentation. Apologies if so.

mourner · 2016-08-18T15:19:27Z

@hdrdavies currently no plans to do this for the HTML markers — this would be a non-trivial effort. Perhaps someone will contribute a plugin.

lucaswoj · 2016-08-18T17:10:35Z

Thanks @mourner. @hdrdavies, you might be able to write your own HTML clustering code using the same backend as GL JS https://github.com/mapbox/supercluster

anandthakker mentioned this pull request Jun 3, 2016

Refactor Source interface to allow/prepare for extensibility #2648

Closed

7 tasks

jfirebaugh reviewed Jun 3, 2016
View reviewed changes

mollymerp mentioned this pull request Jun 7, 2016

Property merging or calculated values #2671

Closed

anandthakker reviewed Jun 7, 2016
View reviewed changes

anandthakker force-pushed the source-refactor branch from c1c041d to 2773f2f Compare June 7, 2016 19:09

anandthakker mentioned this pull request Jun 9, 2016

Implement cross-source symbol placement #2703

Closed

Lucas Wojciechowski added 2 commits July 20, 2016 13:07

Hide custom source classes / interfaces from documentation

c991549

We should validate the custom source architecture and build some demo sources before documenting the internals publicly.

Namespace mapbox-gl-test-suite files in debug server

ad2e938

lucaswoj force-pushed the source-refactor branch from 39b7711 to ad2e938 Compare July 20, 2016 20:07

Lucas Wojciechowski added 3 commits July 20, 2016 13:27

Add source docs

54284d8

Remove GeoJSONSource, VideoSource, ImageSource from the public interface

030647d

Minor GeoJSONSource simplification

9f434bd

lucaswoj assigned lucaswoj and unassigned lucaswoj Jul 20, 2016

lucaswoj merged commit def9cb5 into master Jul 20, 2016

lucaswoj deleted the source-refactor branch July 20, 2016 21:44

lucaswoj mentioned this pull request Jul 22, 2016

Fix failing unit test #2902

Merged

anandthakker mentioned this pull request Jul 29, 2016

Pluggable API for third-party geospatial formats #929

Closed

mourner mentioned this pull request Jul 29, 2016

Custom sources #2920

Open

lucaswoj mentioned this pull request Jul 29, 2016

Mock module dependencies in unit tests #2685

Closed

lucaswoj mentioned this pull request Aug 10, 2016

Fix bug preventing attribution from being displayed #2972

Merged

6 tasks

anandthakker mentioned this pull request Aug 11, 2016

Restore worker-side "redo placement" #2983

Closed

This was referenced Aug 21, 2016

Allow user to set headers for XHR vector tile get #2918

Closed

Documentation for SourceCache is outdated #3054

Closed

displaying geojson tiles from a remote server #2120

Closed

jfirebaugh mentioned this pull request Aug 31, 2016

Image layer disappears while zooming #3010

Closed

lucaswoj mentioned this pull request Sep 19, 2016

Fix disappearing zoomed image layers #3222

Merged

3 tasks

anandthakker mentioned this pull request Oct 26, 2016

Problem re-initialising map in Microsoft browsers #3040

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Source interface to allow/prepare for extensibility #2667

Refactor Source interface to allow/prepare for extensibility #2667

anandthakker commented Jun 3, 2016 •

edited by lucaswoj

Loading

jfirebaugh Jun 3, 2016

anandthakker Jun 3, 2016

jfirebaugh commented Jun 3, 2016

jfirebaugh commented Jun 3, 2016

jfirebaugh Jun 3, 2016 •

edited

Loading

lucaswoj Jun 3, 2016

anandthakker Jun 4, 2016

mourner commented Jun 7, 2016

anandthakker commented Jun 7, 2016

anandthakker Jun 7, 2016

anandthakker commented Jun 9, 2016

anandthakker commented Jun 9, 2016

anandthakker commented Jul 20, 2016

mourner commented Jul 22, 2016

lucaswoj commented Jul 22, 2016

hdrdavies commented Aug 17, 2016

mourner commented Aug 17, 2016

hdrdavies commented Aug 18, 2016

mourner commented Aug 18, 2016

lucaswoj commented Aug 18, 2016

Refactor Source interface to allow/prepare for extensibility #2667

Refactor Source interface to allow/prepare for extensibility #2667

Conversation

anandthakker commented Jun 3, 2016 • edited by lucaswoj Loading

jfirebaugh Jun 3, 2016

Choose a reason for hiding this comment

anandthakker Jun 3, 2016

Choose a reason for hiding this comment

jfirebaugh commented Jun 3, 2016

jfirebaugh commented Jun 3, 2016

jfirebaugh Jun 3, 2016 • edited Loading

Choose a reason for hiding this comment

lucaswoj Jun 3, 2016

Choose a reason for hiding this comment

anandthakker Jun 4, 2016

Choose a reason for hiding this comment

mourner commented Jun 7, 2016

anandthakker commented Jun 7, 2016

anandthakker Jun 7, 2016

Choose a reason for hiding this comment

anandthakker commented Jun 9, 2016

anandthakker commented Jun 9, 2016

anandthakker commented Jul 20, 2016

mourner commented Jul 22, 2016

lucaswoj commented Jul 22, 2016

hdrdavies commented Aug 17, 2016

mourner commented Aug 17, 2016

hdrdavies commented Aug 18, 2016

mourner commented Aug 18, 2016

lucaswoj commented Aug 18, 2016

anandthakker commented Jun 3, 2016 •

edited by lucaswoj

Loading

jfirebaugh Jun 3, 2016 •

edited

Loading