[core] Modularize FileSource codebase (#15768)

* [core] Introduce FileSourceManager and use it for default platform impl - Add `FileSourceManager` interface that provides access to `FileSource` instances and means of registering / unregistering `FileSource` factories - Split `DefaultFileSource` into smaller parts - Add `DatabaseFileSource` interface and it's default implementation - Remove inter-dependencies between concrete `FileSource` classes * [build] Add files to next build system * [core] Add generic property setters / getters * [core] Remove setOnlineStatus from OnlineFileSource interface * [core] Hide threading implementation details from DatabaseFileSource interface * [core] Make DB file source methods virtual * [core] Add documentation for DatabaseFileSource and rename one method * [core] Use simple callback instead of ActorRef * [core] Remove ActorRef from OnlineFileSource public header * [core] Add callback to FileSource::forward async API * [core] Pass OfflineRegionDefinition by value * [core] Update tests to use modular file sources * [core] Update unit tests * [core] Update unit tests after rebase * [core] Backport low prio fix for cached requests * [core] Backport pack database * [core] Return removed factory from unRegisterFileSourceFactory * [core] Rename shadowed args in onlinefilesource * [core] Remove simple std::function callback aliases * [core] Expose online file source property keys in public header file * [test-runner] Add proxy file source test runner * [cache] Update mbgl-cache utility to use new file source * [metrics] Rebaseline binary size metrics * [offline] Update offline utility * [core] Update changelog
mapbox · Jan 13, 2020 · 879c44f · 879c44f
1 parent 86a3605
commit 879c44f
Show file tree

Hide file tree

Showing 85 changed files with 1,877 additions and 1,345 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,7 +2,8 @@
 
 ## Master
 
-### New features
+### ✨ New features
+
 - [core] Port line-sort-key and fill-sort-key ([#15839](https://github.com/mapbox/mapbox-gl-native/pull/15839))
 
   The new feature allows to sort line and fill layer features. Similar to `symbol-sort-key`.
@@ -27,7 +28,8 @@
 
   This patch introduces batch conversion between LatLng and ScreenCoordinate in Gl-Native core, so for multiple conversions with single point/latLng previously now it can be done with invoking one function call by passing vector of points/latLngs.
 
-### Bug fixes
+### 🐞 Bug fixes
+
 - [core] Stable position of labels at tile borders in tile mode ([#16040](https://github.com/mapbox/mapbox-gl-native/pull/16040))
 
   These changes allow to avoid cutting-off labels on tile borders if the variable text placement is enabled.
@@ -78,7 +80,8 @@
 
   This fixes rendering by account for the 1px texture padding around icons that were stretched with icon-text-fit.
 
-### Performance improvements
+### 🏁 Performance improvements
+
 - [core] Calculate GeoJSON tile geometries in a background thread ([#15953](https://github.com/mapbox/mapbox-gl-native/pull/15953))
 
   Call `mapbox::geojsonvt::GeoJSONVT::getTile()` in a background thread, so that the rendering thread is not blocked.
@@ -97,11 +100,22 @@
 
   Before this fix, repeated request for an already obtained image was erroneously treated as pending, and it prevented from the tiles load completion.
 
-### Architectural changes
-- [core] Remove Map::cycleDebugOptions ([#16005](https://github.com/mapbox/mapbox-gl-native/pull/16005))
-
-  This function was mostly used by the Android API, which is no longer necessary.
+### 🧩  Architectural changes
 
 - [core] Merge style::Layer::set{Layout,Paint}Property ([#15997](https://github.com/mapbox/mapbox-gl-native/pull/15997))
 
 - [core] Use expected.hpp from mapbox-base ([#15898](https://github.com/mapbox/mapbox-gl-native/pull/15898))
+
+##### ⚠️  Breaking changes
+
+- [core] Refactor DefaultFileSource codebase ([#15768](https://github.com/mapbox/mapbox-gl-native/pull/15768)) 
+  - Adds `FileSourceManager` interface that provides access to `FileSource` instances and means of registering / unregistering `FileSource` factories
+  - Splits `DefaultFileSource` into smaller parts
+  - Adds `DatabaseFileSource` interface and it's default implementation
+  - Removes inter-dependencies between concrete `FileSource` classes
+  - All sources operate on dedicated thread, except `MainResourceLoader` that acts as a dispatcher and works on thread that requested it.
+  - Removes `ResourceOptions::withCacheOnlyRequestsSupport` method
+
+- [core] Remove Map::cycleDebugOptions ([#16005](https://github.com/mapbox/mapbox-gl-native/pull/16005))
+
+  This function was mostly used by the Android API, which is no longer necessary.
diff --git a/bin/cache.cpp b/bin/cache.cpp
@@ -1,5 +1,6 @@
-#include <mbgl/storage/default_file_source.hpp>
+#include <mbgl/storage/file_source_manager.hpp>
 #include <mbgl/storage/resource.hpp>
+#include <mbgl/storage/resource_options.hpp>
 #include <mbgl/util/chrono.hpp>
 #include <mbgl/util/run_loop.hpp>
 
@@ -89,9 +90,9 @@ int main(int argc, char* argv[]) {
     }
 
     mbgl::util::RunLoop loop;
-    mbgl::DefaultFileSource fileSource(args::get(cacheValue), ".");
-
-    fileSource.put(resource, response);
-
+    auto dbfs = mbgl::FileSourceManager::get()->getFileSource(
+        mbgl::FileSourceType::Database, mbgl::ResourceOptions().withCachePath(args::get(cacheValue)));
+    dbfs->forward(resource, response, [&loop] { loop.stop(); });
+    loop.run();
     return 0;
 }
diff --git a/bin/offline.cpp b/bin/offline.cpp
@@ -3,7 +3,9 @@
 #include <mbgl/util/string.hpp>
 #include <mbgl/util/geojson.hpp>
 
-#include <mbgl/storage/default_file_source.hpp>
+#include <mbgl/storage/database_file_source.hpp>
+#include <mbgl/storage/file_source_manager.hpp>
+#include <mbgl/storage/resource_options.hpp>
 
 #include <args.hxx>
 
@@ -161,17 +163,16 @@ int main(int argc, char *argv[]) {
 
 
     util::RunLoop loop;
-    DefaultFileSource fileSource(output, ".");
-    std::unique_ptr<OfflineRegion> region;
+    std::shared_ptr<DatabaseFileSource> fileSource =
+        std::static_pointer_cast<DatabaseFileSource>(FileSourceManager::get()->getFileSource(
+            FileSourceType::Database,
+            ResourceOptions().withAccessToken(token).withBaseURL(apiBaseURL).withCachePath(output)));
 
-    fileSource.setAccessToken(token);
-    fileSource.setAPIBaseURL(apiBaseURL);
+    std::unique_ptr<OfflineRegion> region;
 
     if (inputDb && mergePath) {
-        DefaultFileSource inputSource(*inputDb, ".");
-        inputSource.setAccessToken(token);
-        inputSource.setAPIBaseURL(apiBaseURL);
-
+        DatabaseFileSource inputSource(ResourceOptions().withCachePath(*inputDb));
+
         int retCode = 0;
         std::cout << "Start Merge" << std::endl;
         inputSource.mergeOfflineRegions(*mergePath,  [&] (mbgl::expected<std::vector<OfflineRegion>, std::exception_ptr> result) {
@@ -193,13 +194,15 @@ int main(int argc, char *argv[]) {
 
     class Observer : public OfflineRegionObserver {
     public:
-        Observer(OfflineRegion& region_, DefaultFileSource& fileSource_, util::RunLoop& loop_, mbgl::optional<std::string> mergePath_)
+        Observer(OfflineRegion& region_,
+                 std::shared_ptr<DatabaseFileSource> fileSource_,
+                 util::RunLoop& loop_,
+                 mbgl::optional<std::string> mergePath_)
             : region(region_),
-              fileSource(fileSource_),
+              fileSource(std::move(fileSource_)),
               loop(loop_),
               mergePath(std::move(mergePath_)),
-              start(util::now()) {
-        }
+              start(util::now()) {}
 
         void statusChanged(OfflineRegionStatus status) override {
             if (status.downloadState == OfflineRegionDownloadState::Inactive) {
@@ -215,14 +218,11 @@ int main(int argc, char *argv[]) {
                 bytesPerSecond = util::toString(status.completedResourceSize / elapsedSeconds);
             }
 
-            std::cout << status.completedResourceCount << " / " << status.requiredResourceCount
-                      << " resources"
-                      << status.completedTileCount << " / " << status.requiredTileCount
-                      << "tiles"
-                      << (status.requiredResourceCountIsPrecise ? "; " : " (indeterminate); ")
+            std::cout << status.completedResourceCount << " / " << status.requiredResourceCount << " resources | "
+                      << status.completedTileCount << " / " << status.requiredTileCount << " tiles"
+                      << (status.requiredResourceCountIsPrecise ? " | " : " (indeterminate); ")
                       << status.completedResourceSize << " bytes downloaded"
-                      << " (" << bytesPerSecond << " bytes/sec)"
-                      << std::endl;
+                      << " (" << bytesPerSecond << " bytes/sec)" << std::endl;
 
             if (status.complete()) {
                 std::cout << "Finished Download" << std::endl;
@@ -239,7 +239,7 @@ int main(int argc, char *argv[]) {
         }
 
         OfflineRegion& region;
-        DefaultFileSource& fileSource;
+        std::shared_ptr<DatabaseFileSource> fileSource;
         util::RunLoop& loop;
         mbgl::optional<std::string> mergePath;
         Timestamp start;
@@ -248,24 +248,26 @@ int main(int argc, char *argv[]) {
     static auto stop = [&] {
         if (region) {
             std::cout << "Stopping download... ";
-            fileSource.setOfflineRegionDownloadState(*region, OfflineRegionDownloadState::Inactive);
+            fileSource->setOfflineRegionDownloadState(*region, OfflineRegionDownloadState::Inactive);
         }
     };
 
     std::signal(SIGINT, [] (int) { stop(); });
 
-    fileSource.createOfflineRegion(definition, metadata, [&] (mbgl::expected<OfflineRegion, std::exception_ptr> region_) {
-        if (!region_) {
-            std::cerr << "Error creating region: " << util::toString(region_.error()) << std::endl;
-            loop.stop();
-            exit(1);
-        } else {
-            assert(region_);
-            region = std::make_unique<OfflineRegion>(std::move(*region_));
-            fileSource.setOfflineRegionObserver(*region, std::make_unique<Observer>(*region, fileSource, loop, mergePath));
-            fileSource.setOfflineRegionDownloadState(*region, OfflineRegionDownloadState::Active);
-        }
-    });
+    fileSource->createOfflineRegion(
+        definition, metadata, [&](mbgl::expected<OfflineRegion, std::exception_ptr> region_) {
+            if (!region_) {
+                std::cerr << "Error creating region: " << util::toString(region_.error()) << std::endl;
+                loop.stop();
+                exit(1);
+            } else {
+                assert(region_);
+                region = std::make_unique<OfflineRegion>(std::move(*region_));
+                fileSource->setOfflineRegionObserver(*region,
+                                                     std::make_unique<Observer>(*region, fileSource, loop, mergePath));
+                fileSource->setOfflineRegionDownloadState(*region, OfflineRegionDownloadState::Active);
+            }
+        });
 
     loop.run();
     return 0;