Android power drain fix + heatmap phone screen + shorter/longer!!

The primary motivation for this release is to fix an android power bug in which we detected a trip end after 5 hours instead of 5 minutes. This was an immediately noticeable impact on battery life for both me and my husband, and shows how important it is to do nothing well.

The other big change is to bring the aggregate visualizations to the phone. This is both a proof of concept on both providing functionality to users before we start tracking, and on the effort required to port server side visualizations to the phone (very easy).

And finally, thanks to @krisma, we finally have working comparisons to common trips (before/after and larger/smaller). I don't think that the common trip generation is currently very accurate, so the numbers may not make that much sense, but once we fix the analysis on the server side, they magically should.

Changelog:

• Copy the heatmap code from the server and get it to work
• Add arrow for longer/shorter
• Add clocks for earlier/later
• Fix spacing between cards in the list view
• Stop restarting collection on reboot
• Change the default tripEndStationaryMins to 5

User configuration of collection parameters + detail screen

After a short hiatus, we're back with a new release that addresses two main areas:

• User focused: Thanks to some awesome work by @krisma, we have almost all the parts of @ericpai's design for the diary implemented. In particular:
  • we have the detail views finally working - if you click on a trip, it will now expand to a zoomable map, and also show all the sections for that trip. So if your trip was bike -> bus -> bike, you can see the three parts in three different colors, and also see when you transitioned between them.
  • the map has been changed to be much more attractive, including ensuring that the start and end markers don't make it obscure
  • we have percentages for the modes in both the overall and detail views, so that you can see at a glance whether a walk + motorized trip was a little walking and a lot of motorized transport, or vice versa
  • we have initial integration with common trips - if a trip "common", it will be marked with a yellow star. we will be improving our integration with common trips fairly soon.
  • we've moved to the newest version of cordova-ios component, which fixes a bug with startup. No more WSODs!
• Configuration focused: As part of the research component, I've added the ability to customize the data collection to your needs. In particular:
  • we finally have the big red button and the big green button to turn tracking off and on. If you turn tracking off and then forget to turn it on, we currently don't prompt you to do so, although we probably will at some point.
  • it is possible to configure collection setting parameters, namely the accuracy and the sampling rate, to reduce the power drain. So if you have had issues with power drain, even with geofencing/visit detection turned on, you can configure the collection further to lower the power drain.
    • On iOS, you do this by changing the accuracy - my testing shows that the power drain for "best" and "10 meters" accuracy are almost identical, but "100 meters" accuracy makes a huge difference. Changing the filterDistance does NOT seem to help. Feel free to run your own experiments and tell me what you find!
    • On android, you can do this by changing both the accuracy and/or the filterTime. My tests show that balanced accuracy at 1 sec intervals has roughly the same drain as high accuracy at 30 secs. The default is already high accuracy at 30 secs, so the best way to lower drain would be to move to medium accuracy at 30 secs.

Full list of fixes/changes:

Cross-platform

• Move storage of battery data to the timeseries
• Add battery status collection at other points in the workflow - not just on sync
• Add support for RW documents in the usercache
• Read data collection configuration from a RW document
• Add support for stopping and starting data collection
• Add UI support for changing the data collection parameters
• Significant refactoring to support a diary service that can be used for both the list and detail screens
• Significant refactoring to support a common trip service that can be reused by the list and detail screens to fill in fields

iOS

• upgrade to new version of cordova-ios -> WSOD fixed in all cases

visit notification on iOS + very preliminary common trips UI + trip list fixes

The four big changes in this release are:

• We finally move to visit notification on iOS. This has been working reliably for me since Tue night, and the logs that I got indicated that we were receiving visit notifications even for the people who don't get remote pushes so I am optimistic that it will work for others as well. This should result in a general improvement in battery life across the board for iOS, since we turn off tracking as soon as the trip ends, and don't wait for the end of the hour. It should result in even more dramatic improvements for those of you who weren't receiving silent push notifications correctly last week. Please try it out and let me know if it works any better.
• After mucho work from @joshzarrabi and some last minute massaging by me, we finally have the common trips pipeline all hooked up. The graph is generated on the server, pushed to the client and viewable via a very preliminary new tab. I am a little underwhelmed with my results - I see a fair number of spurious points - but I'd be interested in seeing what others think. We will enable some more exploration/data visualization on that screen as well once @epai has had a chance to think about the design.
• The alert that popped up every time we launched the app is now gone because I made significant progress towards fixing the javascript White Screen of Death issues on iOS. Now, if the javascript starts loading, it completes. There is still an intermittent issue caused by a bug in the cordova platform, which will be fixed in the next cordova-ios platform release. Thanks to this, I have now removed the alert while launching the app.
• Thanks to some hard work by @krisma, we have a better, more functional trip list screen. He tried very hard to get the trip detail screen to work as well, but has been having some issues with angular. We'll try to get that done sometime next week.

Full list of changes:

Cross-platform

• Finally fix WSOD
• Very preliminary common trips screen
• Removed auto-refresh of the diary screens when we have a remote sync since iOS doesn't load all plugins on a non-UI init
• Add ability to email userCache as well

iOS

• Use visit notifications for detecting trip start and end

Android

• Restore ability to email logs on android (done)
• Fixed newly discovered issue with android sync. Basically, the app was crashing every time we tried to read data from the usercache to push to the server. Problem was a combination of long -> double change for timestamps + correct parantheses for query. Required some crazy debugging to fix.

first peek at new UI + iOS sync improvements

The three big changes for this release are:

• Thanks to some awesome work by @krisma, based on thoughtful design by
  @epai, we have a new trip list UI. This is strictly less functional than the
  the previous UI because it doesn't not have as much detail on the sections,
  but we wanted to give everybody a sneak peek to get some early feedback.
  • This is a variation of Eric's design from the SDB meeting with maps added.
    Our instant focus group felt that the maps made it easier to recall what
    had happened than just the textual place and the time since a picture is worth a
    thousand words. Thoughts?
  • Next week, we plan to add a trip detail screen that will show more details
    about the sections, have zoomable maps, and hopefully add some comparisons
    to your stats. That should restore and exceed the functionality of the
    current version, but with a MUCH more usable UI.
• This release should also make Randy happy since it removes the popup on zero trips.
• I finally fixed iOS sync. At least on some phones, in spite of remote pushes
  being received correctly, data was not really pushed to the server. After a
  day spent focusing entirely on this, I finally have it all working, at least
  on a test phone that reliably had the problem earlier. iOS users, do you
  notice that the syncs are working properly without force syncing?
• I finally fixed an android issue with getting the last location that I had
  deferred for a while. Our three main android testers haven't run into it yet,
  but two others have and I thought it was time to make it more robust and
  bring it in line with the iOS code.

Full list of changes:

Cross-platform

• New version of the trip diary screen thanks to @krisma (0cdcccc)
• Removal of the no trips popup, as requested by Randy (488308b)
• Reload the diary when the sync completes to ensure that the cached view is updated (488308b)

iOS

• Fix the sync!
  • A token refresh is pre-emptively generated on every remote push notification so that it can run in parallel with the message creation and increase the chances that the refresh will finish within 30 seconds (shankari/e-mission-data-collection@c217577)
  • Switching to the new JWT library, and to the new session fetcher library makes our remote calls actually run properly in the background (e-mission/cordova-jwt-auth@12f96c6, e-mission/cordova-server-communication@4496bc0)

Android

• Read the location to create the geofence using a new location tracking request and a new intent service. This fixes issue e-mission/e-mission-data-collection#65 - the case that the lastLocation is NULL, which has been reported twice so far, by @changun and @hariv (shankari/e-mission-data-collection@28786c7)
• Fix issue with restarting tracking on an upgrade. Since we had the version check and the initialize code in the plugin_init, it would not be invoked until the user launched the app, which may take a while after the upgrade is complete. Moving it to the sync code ensures that we start tracking again almost immediately, with no user intervention (shankari/e-mission-data-collection@5544afd)

Beautiful onboarding + fix android sync breakage

Cross-platform

• Beautified on boarding screens courtesy @krisma. If you want to try them out, uninstall the app and reinstall
• Automatically truncate logs - only retain data for the past month
• Content-Security-Policy so that android stops nagging us

Android

• Fix android sync breakage

iOS

• Revisit initialize of trip diary state machine and see if it helps with WSOD and remote push issues. It seems to help a bit with WSOD, but not enough

iOS background sync fixes + load screen

Changelog:

• Convert NSLog statements from the communication code to NotificationHelper so that we can see them in our logs
• Fix iOS background sync by using the Bolts framework to launch all the sync operations in parallel
• Change all calls to backgroundSync to use new BFTask instead. This allows us to correctly mark tasks as done.
• Change some stat values on ios to reflect real values
• Change the javascript calls to get and set personal data to also go through native code. This fixes the token refresh issue on iOS, and also
• Initialize android usercache on startup so that we can avoid the javascript usercache table not found error
• Move the server sync init code to pluginInitialize
• Move the startupInit code on android to the plugin initialize instead
• Attempt to improve WSOD issues by delaying bootstrap until the device is ready and removing all resolve dependencies on the root
• Fix android issue with re-initializing the finite state machine on launch
• Handle re-initializing the FSM on upgrades
• Minor logging fix to show the length of the point arrays instead of the points since they are not human readable
• Use the correct directory for the log on iOS so that we can now get emailed bug reports on iOS as well
• Turn on loading messages for various stages of the diary

Alpha version for use by the development team alone

Let's start rolling those releases!

Known issues

In general, there are issues during startup - you need to fiddle/wait a bit for everything to get initialized properly.

I chose not to optimize this further since we are considering moving to a different flow for onboarding anyway, where we defer the login until the user has had a chance to see the aggregate data and decide whether they want to track their own.

Android

• The "login to google" button takes a while to launch the selection dialog, please be patient
• At the end of the launch process, you may get an error about the userCache table not being present.
  • Ignore it, go to the settings and do a force sync.
  • Then, force kill the app and relaunch. After that, everything is should be fine.

iOS

• Immediately on launch, you will get a prompt to turn allow the use of notifications and location tracking. Accept them.
• On every launch, you will see an alert "Time to set breakpoints". Just hit OK. Without that alert, I fairly reliably (p > 0.75) get a WSOD. With the alert, I don't (p = 1).
• There are some issues with refreshing the OAuth token from the main screen. In general, this should not be an issue because the trips are downloaded and cached locally. However, if you don't see trips where you expect to see them and want to force a refresh - go to the "Controls" screen and "Force Sync", which will refresh the token.

The initial 0.0.1 version had so many issues with the WSOD that I ended up making a new version just for iOS. I don't want to merge it to master because the changes were fairly hacky and it is not clear if they are the best way to fix the problem. So this release has two iOS releases - build-1 and build-2.