AwareDev

Server Todo

• Set up experience sampling.
• Use the MQTT connection

Clients To-do

Android - misc

• (d) Can the remote configuration change the upload frequency when subject is in a study?
• When the AWARE Android app was updated via Google Play, it lost all configuration settings.

Android - crashing

Debugging info:

• Go to settings -> developer options -> running services. Is "Applications" a running service? This is the accessibility service and I think it triggers the uploads

Android - security and privacy

• (d) Verify certificate checksum when first getting it - add sha256 to the QRcode.
• (d) does AWARE ping oulu server always, regardless of settings (aware-core/src/main/java/com/aware/Aware.java, line 303)
• (d) How to delete the /sdcard/AWARE folder when app uninstalled?
  • In general, things should be reset back to the previous state once app is uninstalled.
  • Could we change to use /sdcard/data/Android/com.aware.phone/ for the databases, if some option is set?
• Pluggable/configurable hash algorithms
• AWARE has "automatic plugin downloading". We need to check that this is secure and can't be triggered insecurely...

Android - web API

• (d) Customize notification text to show the study name, if available.
• aware-core/src/main/java/com/aware/utils/WebserviceHelper.java lines 288
  • This is the condition for cleaning local databases. Should this be or, instead of and ?
• Data is not cleaned for all probes
  • Currently this is only immediately cleaned for high-frequency probes.
  • iOS can delete all data immediately regardless of the table type.
• Uploads make too many requests if there is nothing to do.

Android - sensors

• Add step counter probe
• Bluetooth beacons
  • Do they work if app is not running in the background?
• Ambient noise

Android - low priority

• (d) Can config turn off sensors that were on before?

iOS to-do

• Needs to be linkable to other servers
• Do bluetooth beacons work if the app is not running in the background?
• Is data deleted as soon as it is uploaded?

Finished To-do

• clearing app data, or even completely uninstalling the app, does not
  • solution: remove sdcard AWARE/ folder.
• the first POST sometimes causes things to crash. I think this may be somehow related to SSL cert verification. It uses the same cert for https and MQTT.
• Can we make it so that people in a study can edit their settings?
  • yes, set study_id="" (null string) and then data can be sent but config is not updated.
• Locking remote config so it can't be changed.
  • soft-locking: set suggestions, allow to be overridden (this is done when study_id="")
  • hard locking: display active things clearly and why it is set this way. (this is done when study_id="$int", but then the user can not see the settings.)
• Would a persistent notification help on Android? What about wake locks?
• (not done, postponed) https://github.com/denzilferreira/aware-client/blob/master/aware-core/src/main/java/com/aware/utils/SSLManager.java#L36
• This fails if the register URL does not contain /index.php right after the hostname.
• Also get_study_info hardcodes a URL.
• Android app does not run on tablets. (requires back-facing camera)
• Too much data is sent at once: split the data if needed, client or server side.
  • Currently a problem for Light Probe and accelerometer. What other probes does it happen for?
  • Finished by splitting on the server.
• When you enter config URL, you must exit the app and restart before the UI gets updated with all the remote configuration values.
• (d) Show but don't allow edits to configuration when it is locked.
• General: it doesn't long-term sync. This point collects info on that.
  • rkd: When first linked, it synced to battery_charges/latest, after deactivating study and reactiviting (cycling check boxes), it goes all the way to esms/latest and has notification.
• Why do MQTT connections seem sporadic? (some devices always connect, some do not.)
• Android: if AWARE is force-stopped, then it is no longer an accessibility service.
• aware-core/src/main/java/com/aware/utils/WebserviceHelper.java line 288
  • This deletes data after it is uploaded. Does it delete if the POST failed?
• (d) Upload notifications do not remove themselves. aware-core/src/main/java/com/aware/utils/WebserviceHelper.java line 64 for the method that starts them.
  • Maybe they do remove themselves if upload is successful. When sync notifications do not go away, does the app crash or stop uploading?
• Upload happens at fixed intervals on the hour. Randomize this to even out server load.
  • aware-core/src/main/java/com/aware/Aware.java line 420-
• Uploads are hard coded at certain minutes of the hour, this doesn't work well with non-deterministic background running.

Old (upstream) API examples

Get study info.  $study_key is the last segment (last `/` to end) of the config URL.

GET https://api.awareframework.com/index.php/webservice/client_get_study_info/$study_key

response = {"study_name":"Test study","study_description":"Testing of AWARE API","researcher_first":"$firstname","researcher_last":"$name","researcher_contact":"$email"}


================
Get config.  This is what comes from the QR code or what is entered manually as AWARE study.

POST https://api.awareframeworkbservice/index.php/webservice/index/$study_id/$study_key
body = b'device_id=$device_uid'

body (is actually compacted) =
[{"sensors":[{"setting":"status_mqtt","value":"true"},
{"setting":"mqtt_server","value":"api.awareframework.com"},
{"setting":"mqtt_port","value":"8883"},
{"setting":"mqtt_keep_alive","value":"600"},
{"setting":"mqtt_qos","value":"2"},
{"setting":"status_esm","value":"true"},
{"setting":"mqtt_username","value":"$device_uuid"},
{"setting":"mqtt_password","value":"$password"},
{"setting":"study_id","value":"$study_id"},
{"setting":"study_start","value":"1465849409281"},
{"setting":"webservice_server","value":"https:\/\/api.awareframework.com\/index.php\/webservice\/index\/$study_id\/$study_key"},
{"setting":"status_webservice","value":"true"}]}] 

If len(study_id) > 0, then user modification is not allowed.

============
Below is API for sending data.  First `/create_table` is called, then `/latest`, then (if there is new data since the timestamp in /latest) `/insert`.  These three are all called on every sync.

POST https://api.awareframework.com/index.php/webservice/index/$study_id/$study_key/aware_device/create_table
body = fields=_id%20integer%20primary%20key%20autoincrement%2Ctimestamp%20real%20default%200%2Cdevice_id%20text%20default%20''%2Cboard%20text%20default%20''%2Cbrand%20text%20default%20''%2Cdevice%20text%20default%20''%2Cbuild_id%20text%20default%20''%2Chardware%20text%20default%20''%2Cmanufacturer%20text%20default%20''%2Cmodel%20text%20default%20''%2Cproduct%20text%20default%20''%2Cserial%20text%20default%20''%2Crelease%20text%20default%20''%2Crelease_type%20text%20default%20''%2Csdk%20integer%20default%200%2Clabel%20text%20default%20''%2CUNIQUE%20(timestamp%2Cdevice_id)&device_id=$device_uid

body = 
CREATE TABLE IF NOT EXISTS `aware_device` (`_id` int primary key auto_increment, `timestamp` double default 0, `device_id` varchar(150) default '', `board` text default '', `brand` text default '', `device` text default '', `build_id` text default '', `hardware` text default '', `manufacturer` text default '', `model` text default '', `product` text default '', `serial` text default '', `release` text default '', `release_type` text default '', `sdk` int default 0, `label` text default '', UNIQUE (timestamp,device_id))

At this point the study options appear on the dashboard.  If this method is run again with same arguments, it returns null body.  The body is not used in the AWARE clients.

===============
POST https://api.awareframework.com/index.php/webservice/index/$study_id/$study_key/aware_device/latest
body = device_id=$device_uid

response = []

===============
Example of sending data:

POST https://api.awareframework.com/index.php/bservice/index/$study_id/$study_key/screen/insert
body = "data=$urlencoded_json&device_id=%device_uuid"

response = ""  (null body)

=============
Then doing /latest on same table:

POST  https://api.awareframework.com/index.php/webservice/index/$study_id/$study_key/screen/latest
body = 'device_id=$device_uuid'

response = [{"timestamp":"1465850014988"}]

New API

This section describes our API improvements. It is designed to be backwards compatible, just with some extra options that can make Android more more efficient, safer, and like the iOS client.

Nomenclature: API v1 is the upstream AWARE API. API v1b is our group's version.

New options in API v1b:

• webservice_only (default false): Means that data is only sent to the server, and is not supposed to be ever stored on the device long-term.
• webservice_stateless (default false): Means that we don't have a two-stage process of /latest and /insert, and we don't do any /create_table. The phone has to manage sending data without knowing anything from the server.
• These options are not present in the mobile app UI, and can only be set by the server (see below)

1: create_table: [API v1b: if webservice_stateless is empty/false]

POST {base_url}/{table}/create_table
Body: {some ALTER TABLE stuff}

Response: {some ALTER TABLE stuff}

This sends some information about the current database schema, and the server possibly runs ALTER_TABLE if needed to synchronize schemas.

• API v1: Makes the request before syncing data.
• API v1b: If webservice_stateless is true, then skip this step entirely. Otherwise, do the same as API v1. With this v1b, the server stores all raw data that is received and it is processed later.

2: /latest [API v1b: if webservice_stateless is empty/false]

POST {base_url}/{table}/latest
Body: null or FormEncode(nonce={random data})

Response:
[ {"timestamp":{unixtime_ms},
   "double_end_timestamp":{unixtime_ms},
   "double_esm_user_answer_timestamp":{unixtime_ms},
   "nonce": {random data}                                [API v1b only]
} ]

This endpoint returns the latest timestamp on the server. It is basically the row in the database with the greatest timestamp. In AWARE, the name of the returned field depends on what table it is (there is explicit conditionals at every point), but in our server, we return the timestamp in analogy to the /insert step. Also, if the request body contains a nonce, then this is returned as part of the row. Otherwise, nonce is not there.

• API v1: Do this step as above.
• API V1b: If webservice_stateless is true, then do not do this step.

3: /insert: [API v1b: Run if (webservice_only is empty/false) or (count(*) > 0). Meaning, if webservice_only is empty, use old behavior. Otherwise, upload data only if data exists.]

POST {base_url}/{table}/insert
Body: FormEncode(data=JSONencode({json}), nonce={random data})
      {json} = [{ row }, { row } ]

Response v1: null

Response v1b:
[ {"timestamp": {unixtime_ms},
   "double_end_timestamp": {unixtime_ms},
   "double_esm_user_answer_timestamp": {unixtime_ms},
   "nonce": {same random data},
} ]

This is the actual data upload method. It POSTS formencoded data, of which the data key is JSON-encoded rows. The nonce is returned if it is given by the request.

• API v1: this data is directly inserted in the database. The Response is null.
• API v1b: If webservice_only is true, then do the below steps. If not, use the default actions above.
  • /insert only run if data exists (count(*) > 0). If no data exists, then do nothing.
  • Since we have no timestamps from /latest, send ALL the available data to the server.
  • Delete ALL data from the local database after it is sent.
  • The server may get duplicate data. Since timestamps are always ascending, and the order of the data is known, the server can exclude duplicate data later.

Comments

• If neither webservice_only or webservice_stateless are given, behavior should be the same as API v1.
• API v1b should be implemented by using several conditionals using these variables, using code which is mostly already existing.
• When these variables are set, the Android behavior should be the same as the iOS behavior.
• These apps are not available in the mobile application UI. One of the design criteria of the upstream AWARE is that data is preserved on the device so that researchers can get it off (even if it is being uploaded...). To maintain upstream compatibility for a possible merge into upstream, these are not presented to the UI, and the server has to enable them.
• The nonce is included to guard against replays and caching. (it is optional, if it is not provided by client the server does nothing). This should not be necesary because TCP + SSL should already do that, SSL cert pinning should protect against intermediate servers, and POST is supposed to never be cached. However, it seemed like the API may have been designed to not fully trust HTTP requests, so this is added as an option if clients want to use it to ensure that requests are trustable. It no effect on the server.

Todo

• What about data collected before the study was started or the device was linked?