Group traffic by referrer

[ro-tex]

Overview

We want to be able to inform users why certain traffic took place, i.e. which skapp downloaded or uploaded a given file. For that we add a referrer value on nginx level. The remaining question is how to organise that on accounts' side.

Details

Normalise the referrer

There are several types of referrers that we can support: HNS domains, skylinks, and regular websites. We should come up with a canonical representation of each and store only that.

Example:
HNS domains can be myskapp.hns.siasky.net or siasky.net/hns/myskapp or any-domain-actually.siasky.net/hns/myskapp/something-something. All of these should be normalised to myskapp.

Question:
How should we normalise skylinks? Use the base64 link? Use the base32 link? Do not normalise?

Question:
How should we normalise regular websites? Fully qualified domain name? FQDM+path?

Changes to existing endpoints

Both GET /user/uploads and GET /user/downloads will get a referrer parameter that filters by referrer. This parameter doesn't need to be normalised, the endpoints will take care of that.

New endpoint

GET /user/traffic will report on the top referrers (by volume) for the last [period]. Both the number of referrers returned and the length of the period will be configurable via parameters (potentially in a F/U). The endpoint will also return the traffic statistics for those referrers over the last 24 hours.

Just to clarify:
The endpoint returns data on two time ranges. Time range 1 (e.g. 1 week) is the one that defines which referrers are included in the report. Time range 2 is always 24 hours and doesn't affect the selection of referrers.

Returned data:

[
    {
        "source": "skygallery",
        "sourceType": "hns",
        "period": {
            "downloadSize": 123, // bytes?
            "downloadCount": 123,
            "uploadSize": 123,
            "uploadCount": 123,
            "registryReads": 123,
            "registryWrites": 123
        },
        "last24Hours": {
            "downloadSize": 123,
            "downloadCount": 123,
            "uploadSize": 123,
            "uploadCount": 123,
            "registryReads": 123,
            "registryWrites": 123
        }
    },
    ...
]