Skip to content

Live Stats

Jump to bottom Edit New page
thibaudgg edited this page Oct 2, 2012 · 14 revisions

!!! NO MORE UP-TO-DATE !!!

Implementation roadmap

  1. CDN-Based site stats automatically updated (push) every minute. No true live stats and no unique viewers. (original)
  2. Add unique visitors/viewers to site stats (original)
  3. Add true Live site stats (current visitors and current viewers) (original)
  4. Video stats (to be announced with video hosting, but can be used even if video not hosted by us) (original)

Notes

  • Period/Range available in popup:
    • Last 60 minutes
    • Last 24 hours
    • Last 7 days
    • Last 30 days
    • All time
    • Custom range
  • The period/range is automatically selected depending on activity (if there're no views in the last hour, we are not going to automatically selected "Last 60 minutes")
  • Browser/OS pie chart is based on Page load data, HTML5/Flash pie chart is based on video prepared data
  • Page load vs. Video load
    • Page load is what we currently call "video pageview".
    • Decided to avoid displaying both (too complicated to understand)
    • So in the site stats we do not show the ratio views/pageload because it doesn't make much sense (if you have multiple videos on 1 page)
    • We'll introduce the "video load" concept on phase 4 (with the video stats). This is what the other video platforms provide: when they say "page load" it is actually a per-video-counter that is incremented every time the video is loaded on the page, we'll call this "video load", and so if you visit a single page with 4 playbable videos in it, you'll generate 1 page load and 4 video loads...

Integration with MSV

  • There is no Stats tab. (Only "Sites" and "Videos" tabs)
  • Sites page - Site row
    • Last 30 days video views count
    • Link to Site stats page
  • Site stats page
    • (see above)
    • sortable table with videos (with paginate) + searchfield (we'll have to find a way to name videos)
    • click on table row to access video stats page
  • Video stats page
    • Title is something like: "Stats for 'VideoX' on 'SiteY'"
    • Type of stats we'll show here are still to be defined precisely, but we'll definitely have stats similar to the ones we present in the Site stats page, but instead of page load, we'll rather have the video load number:

      • of video loads and # of video views

      • ratio views/load (%)
      • chart showing views and loads
      • embed stats (with referrer)
      • engagement chart
      • more to be defined
  • Videos page - Video row
    • Can access to video stats page for a particular site (that will be selected from popup)

Top videos section in site stats

Top X (10) video in the current select period (day/hour/minute/second) sorted by views (at first)

Table fields:

  • Posterframe (with click to play the video with all current sources)
  • Name
    • Info box if name doesn't comes from data-name
    • Info box if token doesn't comes from data-uid
  • Total of views for current period
  • Total of loads for current period
  • Sparkline of views for current period (not displayed for last 60 seconds)

Later:

  • Sort by loads
  • "Show more" if more than 10 videos
  • Search fields (on video name)

GIF request params (For Step 1 & 2)

All params need to be URL encoded (escaped). Maybe truncate HTTP GET request to 2048 characters (IE has a limitation of 2048 char.)

GIF urls:

Global params (must be set for each requests)

Params Descrption Example
t Site token t=23fdsfsf
e Event type (load, start,...) e=l / e=s
i Timestamp to prevent browser caching of the GIF i=1310389135336
h Hostname license type (main, extra, dev or invalid) h=m / h=e / h=d / h=i
d Device (desktop/mobile) d=d / d=m / d=t
em* is iframe Embedded em=1

(*) optional param, nil/false if not present

Note:

  • Send e=load requests after license check

Step 2 only:

Params Descrption Example
u User unique ID (Cookie based) 24x[a-z0-9] u=346t5rd98y98a73ynio8wp98

Page visit / Videos prepare params (e=l)

Parent page is used in case of an embedded (iframe) prepared video. One request can contains mutliple prepared videos info.

Params Descrption Example
po* Video Prepare only (page already visited) po=1
dt (Parent) document title (only non-embed) pt=analytics%20page%20test
du (Parent) document url ph=http%3A//jime.com/testDirectory/myPage.html%3Fwer
ru* (Parent) referrer url (only non-embed) ru=http%3A//www.google.com/search%3Fq%3Dbob
eu* Embedded iframe src url eu=http%3A//cdn.sublimevideo.net/e/123asda3%3Ft%3D23fdsfsf
fv* Flash version fv=9.0%20r48&
sr Screen resolution sr=2400x1920
bl* Browser language bl=pt-br
vu[]** Video uids vu[]=asd123ef
pz[] Player sizes pz[]=400x300
pt[] Player types (normal/lightbox/simple/fake) pt[]=n / pt[]=l / pt[]=s / pt[]=f
pm[] Player modes (html5/flash) pm[]=h / pm[]=f
pff[]* Player modes flash forced pff[]=1

(*) optional param, nil/false if not present

(**) we must strip GET params from the source path/url used to compute this param

Note:

  • Video uid comes from data-uid video tag attribute or from the first video source crc32

Video view params (e=s)

Params Descrption Example
pt Player type (normal/lightbox/simple/fake) pt=n / pt=l / pt=s / pt=f
pm Player mode (html5/flash) pm=h / pm=f
vu** Video uid vu=asd123ef
vuo Video uid origin (attribute/source) vuo=a / vuo=s
vn Video name vn=My%20Awesome%20Video
vno Video name origin (attribute/source) vno=a / vno=s
vs Video source url vs=http%3A//videos.sublimevideo.net/123asda3.mp4
vc** Video source crc32 vc=5062d010
vcs[]** Video crc32 sources array (in order) vcs[]=5062d010
vsq Video source quality (hd/base) vsq=hd / vsq=base
vsf Video source family (mp4/webm/ogg) vsf=mp4 / vsf=webm / vsf=ogg
vsr* Video source resolution vsr=400x300
vsd* Video source duration in ms (can be undefined if live streaming) vsd=12345
vp* Video poster url vp=http%3A//posters.sublimevideo.net/123asda3.png

(*) optional param, nil/false if not present

(**) we must strip GET params from the source path/url used to compute this param

Note:

  • Video uid comes from data-uid video tag attribute or from the first video source crc32
  • Video name comes from data-name video tag attribute or from the first video source filename "maybe humanized" (without the extension)

MongoDB collections & fields

site_stats collection

  • site token (t)
  • seconds (s) / minutes (m) / hours (h) / days (d): DateTime
  • page visits hash (pv) { m (main) => 2, e (extra) => 10, d (dev) => 43, i (invalid) => 2, em (main & extra embed) => 1 }
  • video views hash (vv) { m (main) => 1, e (extra) => 3, d (dev) => 11, i (invalid) => 1, em (main & extra embed) => 1 }

Only counted for main & extra hostname (non-embed)

  • Player mode + device hash of hash (md) { h (html5) => { d (desktop) => 2, m (mobile) => 1 }, f (flash) => ... } (on each video loads)
  • Browser + Plateform hash (bp) { "saf-win" => 2, "saf-osx" => 4, ...} (on page visit)

video_stats collection

  • site token (st)
  • video uid (u)
  • seconds (s) / minutes (m) / hours (h) / days (d): DateTime
  • video loads hash (vl) { m (main) => 2, e (extra) => 10, d (dev) => 43, i (invalid) => 2, em (main & extra embed) => 1 }
  • video views hash (vv) { m (main) => 1, e (extra) => 3, d (dev) => 11, i (invalid) => 1, em (main & extra embed) => 1 }

Only counted for main & extra hostname (non-embed)

  • Player mode + device hash of hash (md) { h (html5) => { d (desktop) => 2, m (mobile) => 1 }, f (flash) => ... } (on video load)
  • Browser + Plateform hash (bp) { "saf-win" => 2, "saf-osx" => 4, ...} (on video load)
  • Video sources views hash (vs) { '5062d010' (video source crc32) => 32, ... } (on video view)

video_tags collection

  • site token (st)
  • video uid (u)
  • video uid origin (uo)
  • video name (n)
  • video name origin (no)
  • video poster url (p)
  • video current sources array (cs) ['5062d010' (video source crc32), 'abcd1234', ... ] # sources actually used in the video tag
  • video sources hash (s) { '5062d010' (video source crc32) => { u (source url) => 'http://.../dartmoor.mp4', q (quality) => 'hd', f (family) => 'mp4', r (resolution) => '320x240' }, ... }
  • timestamps

Note: Video information is updated each minutes when there's a play on one of his video sources and video tag has changed (Pushed!).

Browser Supported

Abbr. Description
fir Firefox
chr Chrome
iex Internet Explorer
saf Safari
and Android
rim BlackBerry
web webOS
ope Opera
oth Other

Platform Supported:

Abbr. Description
win Windows
osx Mac OS X
ipa iOS (iPad)
iph iOS (iPhone)
ipo iOS (iPod)
lin Linux
and Android
rim BlackBerry
web webOS
wip Windows Phone
otd Other (Desktop)
otm Other (Mobile)

Tasks to be done

Step 1

Feature Owner Estimate State
Define gif tracking API (which params to send for wich method) Z, T 1-2 ✓
Optimizing logs fecth/download (without using the super slow voxel.voxcast.ondemand.logs.list method) T 1 ✓
Implement & deploy gif tracking in SV player (to have data to shown in september) Z ?
Finalize wireframe (maybe already done) O, T, Z 1-2 ~✓
Think about MongoDB stats structure (one collection for minutes / hours / days) T 1 ✓
Implement gif log parsing (with MongoDB storage) T 3-4 ✓
Learn Backbone.js T 1-2 ✓
Implement site stats view (with highchart + backbone) T 5-7 ✓
Add automatic data reload (ajax) with Pusher T 2-3 ✓
Design CSS/Photoshop O ?
Total 14-22

Step 2

Feature Owner Estimate State
Add unique user id (UUID) support in SV gif tracking (cookie based) Z ?
Think about how storing unique visitor/viewer (MongoDB vs Redis) T 1-2
Update gif log parsing for unique visitor/viewer T 2-3
Add unique visitor/viewer counter to site stats view (with automatic update) T 2-3
Design CSS O ?
Total 5-8

Step 3

Tasks still need to be clearly identified

  • Think about:
    • how live server (node.js or goliath) work, maybe it doesn't need to store data (in MongoDB or redis) but only parse it and push it to the client.
    • if SV player is always sending live request or only when a sv customer has his site stats page open (in that case all SV player instance used must be connected via websocket to start sending data -> super big number of connection needed))
    • how CDN data (update every minute) are merged with live data
  • Define how and when SV player instance send data (like gosquared certainly or maybe via gif requests served via live server)
  • Implement live server
    • client request parsing
    • storing data in db, maybe not useful
    • push data to msv site stats view (via Puhser)
  • Update backbone code to support live push data
  • Add live data view
  • Update existing data in realtime (live)
  • Design CSS

Competitors examples:

Stats provider via Extension/Addon

Resources:

© 2013 Jilion

Dev

Documentation

Archives

Clone this wiki locally