[Continued] Google Analytics import #1753
Commits on Feb 26, 2022
-
-
-
-
-
-
-
-
Pass another dataset to main graph for rendering in red
This adds another entry to the JSON data returned via the main graph API called `imported_plot`, which is similar to `plot` in form but will be completed with previously imported data. Currently it simply returns the values from `plot` / 2. The data is rendered in the main graph in red without fill, and without an indicator for the present. Rationale: imported data will not continue to grow so there is no projection forward, only backwards.
-
-
-
-
-
Start converting metrics to atoms so they can be subqueried
This changes "visitors" and in some places "sources" to atoms. This does not change the behaviour of the functions - the tests all pass unchanged following this commit. This is necessary as joining subqueries requires that the keys in `select` statements be atoms and not strings.
-
-
-
-
-
-
-
-
-
-
Also add tentative imported_utm_sources table
This needs a bit more work on the google import side, as GA do not report sources and utm sources as distinct things.
-
Return imported data to dashboard for rest of Sources panel
This extends the merge_imported function definition for sources to utm_sources, utm_mediums and utm_campaigns too. This appears to be working on the DB side but something is incomplete on the client side.
-
-
Merge entry pages and exit pages from imported data into unfiltered d…
…ashboard view This requires converting the `"visits"` and `"visit_duration"` metrics to atoms so that they can be used in ecto subqueries.
-
-
-
-
-
-
-
-
-
Get timeOnPage and pageviews for pages from GA
These are needed for the pages modal, and for calculating exit rates for exit pages.
-
-
-
-
-
Parse referrers with RefInspector
- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual referrer host + path, whereas 'ga:source' includes utm_mediums and other values when relevant. - 'ga:fullReferror' does however include search engine names directly, so they are manually checked for as RefInspector won't pick up on these.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Clean locations import from Google Analytics
- Country and region should be set to "" when GA provides "(not set)" - City should be set to 0 for "unknown", as we cannot reliably import city data from GA.
-
-
os -> operating_system in some parts of code
The inconsistency of using os in some places and operating_system in others causes trouble with subqueries and joins for the native and imported data, which would require additional logic to account for. The simplest solution is the just use a consistent word for all uses. This doesn't make any user-facing or database changes.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Clean up GA request code a bit
There was some weird logic here with two separate lists that really ought to be together, so this merges those.
-
-
-
-
-
Reorganise GA data fetch logic
- Fetch data from the start of time (2005) - Check whether no data was fetched, and if so, inform user and don't consider data to be imported.
-
-
Apply location filters from API
This makes it consistent with the sources etc which filter out 'Direct / None' on the API side. These filters are used by both the native and imported data handling code, which would otherwise both duplicate the filters in their `where` clauses.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Split sources and UTM sources from fetched GA data
GA has only a "source" dimension and no "UTM source" dimension. Instead it returns these combined. The logic herein to tease these apart is: 1. "(direct)" -> it's a direct source 2. if the source is a domain -> it's a source 3. "google" -> it's from adwords; let's make this a UTM source "adwords" 4. else -> just a UTM source
-
