Skip to content

5. Extending CARP Mobile Sensing

Jump to bottom
Jakob E. Bardram edited this page Oct 1, 2023 · 39 revisions

The main purpose of the CARP Mobile Sensing framework is to allow for extension with domain-specific studies, tasks, measures, probes, and datums (i.e., data). This is done by implementing classes that inherits from the (abstract) classes in the library.

There are a couple of ways to extend the library according to the following scenarios:

Adding New Sampling Capabilities

If you want to add new sensing capabilities you would basically create a new SamplingPackage. A sampling package implements the following classes:

  • SamplingPackage - the overall specification of what this package does.
  • SamplingSchema - specifies sampling configurations.
  • Data - specifies the data model of the data collected.
  • Probe - implements the runtime of data collection.
  • DeviceManager - specifies how an external device is managed (if any).

Not all sampling packages use an external device and are hence simpler to implement since they can extend the default SmartphoneDeviceManager. Let us use the DeviceSamplingPackage as an example of a sampling package that does not use any external device.

Sampling Package

The first step is that your new sampling package should implement the SamplingPackage interface, including all the instance methods.

class DeviceSamplingPackage extends SmartphoneSamplingPackage {
  /// Measure type for collection of basic device information.
  static const String DEVICE_INFORMATION = '${CarpDataTypes.CARP_NAMESPACE}.deviceinformation';

  /// Measure type for collection of free physical and virtual memory.
  static const String FREE_MEMORY = '${CarpDataTypes.CARP_NAMESPACE}.freememory';

  /// Measure type for collection of battery level and charging status.
  static const String BATTERY_STATE = '${CarpDataTypes.CARP_NAMESPACE}.batterystate';

  /// Measure type for collection of screen events (on/off/unlocked).
  static const String SCREEN_EVENT = '${CarpDataTypes.CARP_NAMESPACE}.screenevent';

  @override
  DataTypeSamplingSchemeMap get samplingSchemes =>
      DataTypeSamplingSchemeMap.from([
        DataTypeSamplingScheme(DataTypeMetaData(
          type: DEVICE_INFORMATION,
          displayName: "Device Information",
          timeType: DataTimeType.POINT,
        )),
        DataTypeSamplingScheme(
            DataTypeMetaData(
              type: FREE_MEMORY,
              displayName: "Free Memory",
              timeType: DataTimeType.POINT,
            ),
            IntervalSamplingConfiguration(
              interval: const Duration(minutes: 1),
            )),
        DataTypeSamplingScheme(DataTypeMetaData(
          type: BATTERY_STATE,
          displayName: "Battery State",
          timeType: DataTimeType.POINT,
        )),
        DataTypeSamplingScheme(DataTypeMetaData(
          type: SCREEN_EVENT,
          displayName: "Screen Events",
          timeType: DataTimeType.POINT,
        )),
      ]);

  @override
  Probe? create(String type) {
    switch (type) {
      case DEVICE_INFORMATION:
        return DeviceProbe();
      case FREE_MEMORY:
        return MemoryProbe();
      case BATTERY_STATE:
        return BatteryProbe();
      case SCREEN_EVENT:
        return (Platform.isAndroid) ? ScreenProbe() : null;
      default:
        return null;
    }
  }

  @override
  void onRegister() {
    FromJsonFactory().register(DeviceInformation());
    FromJsonFactory().register(BatteryState());
    FromJsonFactory().register(FreeMemory());
    FromJsonFactory().register(ScreenEvent());
  }
}

Sampling Configurations

Default sampling configurations can be specified as part of samplingSchemes, like the IntervalSamplingConfiguration specified for the FREE_MEMORY measure type above.

There are a number of built-in sampling configurations available:

But you may need to write your own sampling configuration that is tailored to a specific measure and hence how data should be collected in a probe. As an example of how to write a sampling configuration, the IntervalSamplingConfiguration is shown below:

/// A sampling configuration that allows configuring the time [interval] in
/// between subsequent measurements.
@JsonSerializable(fieldRename: FieldRename.none, includeIfNull: false)
class IntervalSamplingConfiguration extends PersistentSamplingConfiguration {
  /// Sampling interval (i.e., delay between sampling).
  Duration interval;

  IntervalSamplingConfiguration({required this.interval}) : super();

  @override
  Function get fromJsonFunction => _$IntervalSamplingConfigurationFromJson;
  @override
  Map<String, dynamic> toJson() => _$IntervalSamplingConfigurationToJson(this);
  factory IntervalSamplingConfiguration.fromJson(Map<String, dynamic> json) =>
      FromJsonFactory().fromJson(json) as IntervalSamplingConfiguration;
}

Note that a sampling configuration must extend from one of the sampling configuration types (e.g., SamplingConfiguration or PersistentSamplingConfiguration).

Sampling configurations can be part of a study protocol where you can override the sampling configuration of a measure. If you want your own sampling configurations to be serializable to JSON, you need to add JSON serialization according to the carp_serializable package to them. This entails

  • annotated with @JsonSerializable
  • implement the three JSON serializable methods; fromJsonFunction, fromJson, toJson
  • register the configuration's fromJsonFunction in the FromJsonFactory, e.g. in the onRegistry() callback function as shown below:
  @override
  void onRegister() {
    FromJsonFactory()
        .register(IntervalSamplingConfiguration(interval: Duration(days: 1)));
  }

See the carp_serializable package for details on serialization.

Datum

Next, you should create the package-specific Datum objects that the package collects. Here is the example of the collected FreeMemoryDatum;

/// Holds information about free memory on the phone.
@JsonSerializable(fieldRename: FieldRename.snake, includeIfNull: false)
class FreeMemoryDatum extends Datum {
  @override
  DataFormat get format => DataFormat.fromString(DeviceSamplingPackage.MEMORY);

  /// Amount of free physical memory in bytes.
  int? freePhysicalMemory;

  /// Amount of free virtual memory in bytes.
  int? freeVirtualMemory;

  FreeMemoryDatum([this.freePhysicalMemory, this.freeVirtualMemory]) : super();

  factory FreeMemoryDatum.fromJson(Map<String, dynamic> json) => _$FreeMemoryDatumFromJson(json);

  @override
  Map<String, dynamic> toJson() => _$FreeMemoryDatumToJson(this);

  @override
  String toString() => '${super.toString()}, physical: $freePhysicalMemory, virtual: $freeVirtualMemory';
}

Again note that;

  • a datum must always extend Datum (unless you are writing your own custom DataManager).
  • must be annotated with the @JsonSerializable
  • must implement the json serialization methods; fromJson and toJson

A small note on privacy; it is possible to add privacy protection of collected data to the package. See how to create data privacy for how to do this.

Probes

The next step is to implement probes for each measure type. A probe implements how a measure type is collected, collects the data and returns it as datum object(s).

The generic Probe interface explains how to use a probe. But in order to create your own probes, CAMS have a set of predefined, abstract probes to extend from:

The DeviceProbe is an example of a simple DatumProbe that collects device information about the phone using the the device_info_plus Flutter package and map this to a DeviceDatum . As shown below, the only thing this probe does it to implement the getDatum() method, which return the device info mapped to a DeviceDatum.

/// A probe that collects the device info about this device.
class DeviceProbe extends DatumProbe {
  @override
  Future<Datum?> getDatum() async => DeviceDatum(
        DeviceInfo().platform,
        DeviceInfo().deviceID,
        deviceName: DeviceInfo().deviceName,
        deviceModel: DeviceInfo().deviceModel,
        deviceManufacturer: DeviceInfo().deviceManufacturer,
        operatingSystem: DeviceInfo().operatingSystem,
        hardware: DeviceInfo().hardware,
      );
}

The ScreenProbe is an example of a StreamProbe that collects screen activity data by implementing the get stream property, which maps screen events from the screen_state plugin to ScreenDatum datum objects.

/// A probe collecting screen events:
///  - SCREEN ON
///  - SCREEN OFF
///  - SCREEN UNLOCK
/// which are stored as a [ScreenDatum].
///
/// This probe is only available on Android.
class ScreenProbe extends StreamProbe {
  Screen screen = Screen();

  @override
  Stream<Datum>? get stream => (screen.screenStateStream != null)
      ? screen.screenStateStream!
          .map((event) => ScreenDatum.fromScreenStateEvent(event))
      : null;
}

Auto generation of JSON files

Now that all packages classes have been designed, the final steps are to generate the json serialization helper classes (from the carp_serializable Dart package) and to bundle the package together.

The files used in json serialization can be generated using build_runner, by running the following command in the root of your Flutter project:

flutter packages pub run build_runner build --delete-conflicting-outputs

This will generate the necessary .g.dart files.

Putting the package together

Finally, all files should be bundled together in a Dart library. For example, the carp_device_package package looks like this;

/// A library containing a sampling package for collecting information from the device hardware:
///  - device info
///  - battery
///  - screen
///  - free memory
library carp_device_package;

import 'dart:async';
import 'dart:io' show Platform;

import 'package:carp_core/carp_core.dart';
import 'package:carp_mobile_sensing/carp_mobile_sensing.dart';

import 'package:battery_plus/battery_plus.dart';
import 'package:json_annotation/json_annotation.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:screen_state/screen_state.dart';
import 'package:system_info2/system_info2.dart';

part 'carp_device_package.g.dart';
part 'device_datum.dart';
part 'device_package.dart';
part 'device_probes.dart';

To use a sampling package, import it into your app together with the carp_mobile_sensing package:

import 'package:carp_core/carp_core.dart';
import 'package:carp_mobile_sensing/carp_mobile_sensing.dart';
import 'package: carp_device_package/carp_device_package.dart';

Before creating a study and running it, register this package in the SamplingPackageRegistry.

  SamplingPackageRegistry().register(DeviceSamplingPackage());

A package can also be released as a Dart Package on Pub. We already provide a list of different sampling packages - both using the onboard phone sensors as well as external wearable devices and online services.

Device Manager

If a sampling package handles (i.e., collects data from) an external device (or service), then it should be able to specify what type of device it supports and provide corresponding DeviceManager and DeviceDescriptor classes. For example, in the eSense sampling package, the ESenseSamplingPackage implements the following two methods:

class ESenseSamplingPackage implements SamplingPackage {
  static const String ESENSE_DEVICE_TYPE = 'esense';

  // other parts of the package omitted

  // the type of device supported
  String get deviceType => ESENSE_DEVICE_TYPE;

  // a device manager for this device
  DeviceManager get deviceManager => ESenseDeviceManager();
}

The ESenseDevice device descriptor describes how an eSense device is to be configured:

/// A [DeviceDescriptor] for an eSense device used in a [StudyProtocol].
@JsonSerializable(fieldRename: FieldRename.none, includeIfNull: false)
class ESenseDevice extends DeviceDescriptor {
  /// The type of a eSense device.
  static const String DEVICE_TYPE =
      '${DeviceDescriptor.DEVICE_NAMESPACE}.ESenseDevice';

  /// The default rolename for a eSense device.
  static const String DEFAULT_ROLENAME = 'esense';

  /// The name of the eSense device.
  /// Used for connecting to the eSense hardware device over BTLE.
  /// eSense devices are typically named `eSense-xxxx`.
  String? deviceName;

  /// The sampling rate in Hz of getting sensor data from the device.
  int? samplingRate;

  ESenseDevice({
    super.roleName = ESenseDevice.DEFAULT_ROLENAME,
    this.deviceName,
    this.samplingRate,
    super.supportedDataTypes,
  }) : super(
          isMasterDevice: false,
        );

  @override
  Function get fromJsonFunction => _$ESenseDeviceFromJson;
  factory ESenseDevice.fromJson(Map<String, dynamic> json) => FromJsonFactory().fromJson(json) as ESenseDevice;
  @override
  Map<String, dynamic> toJson() => _$ESenseDeviceToJson(this);
}

Note again that since a device descriptor can be part of a study protocol, it needs to support JSON serialization.

The ESenseDeviceManager then implements the DeviceManager interface.

/// A [DeviceManager] for the eSense device.
class ESenseDeviceManager extends HardwareDeviceManager<DeviceRegistration, ESenseDevice> {
  // the last known voltage level of the eSense device
  double? _voltageLevel;

  /// A handle to the [ESenseManager] plugin.
  ESenseManager? manager;

  @override
  String get id => deviceDescriptor?.deviceName ?? 'eSense-????';

  /// A estimate of the battery level of the eSense device.
  @override
  int? get batteryLevel => (_voltageLevel != null)
      ? ((1.19 * _voltageLevel! - 3.91) * 100).toInt()
      : null;

  @override
  void onInitialize(DeviceDescriptor descriptor) {
    if (deviceDescriptor?.deviceName == null ||
        deviceDescriptor!.deviceName!.isEmpty) {
      warning(
          'Cannot initialize an $runtimeType with a null or empty device name. '
          "Please specify a valid device name, typically on the form 'eSense-1234'");
    }
  }

  @override
  Future<bool> canConnect() async => (deviceDescriptor?.deviceName != null &&
      deviceDescriptor!.deviceName!.isNotEmpty);

  @override
  Future<DeviceStatus> onConnect() async {
    if (deviceDescriptor?.deviceName == null ||
        deviceDescriptor!.deviceName!.isEmpty) return DeviceStatus.error;

    manager = ESenseManager(id);

    // listen for connection events
    manager?.connectionEvents.listen((event) {
      debug('$runtimeType - $event');

      switch (event.type) {
        case ConnectionType.connected:
          status = DeviceStatus.connected;

          // this is a hack! - don't know why, but the sensorEvents stream
          // needs a kick in the ass to get started...
          manager?.sensorEvents.listen(null);

          // when connected, listen for battery events
          manager!.eSenseEvents.listen((event) {
            if (event is BatteryRead) {
              _voltageLevel = event.voltage ?? 4;
            }
          });

          // set up a timer that asks for the voltage level
          Timer.periodic(const Duration(minutes: 2), (_) {
            if (status == DeviceStatus.connected) {
              manager?.getBatteryVoltage();
            }
          });
          break;
        case ConnectionType.unknown:
          status = DeviceStatus.unknown;
          break;
        case ConnectionType.device_found:
          status = DeviceStatus.paired;
          break;
        case ConnectionType.device_not_found:
        case ConnectionType.disconnected:
          status = DeviceStatus.disconnected;
          _voltageLevel = null;
          // _eventSubscription?.cancel();
          break;
      }
    });

    manager?.connect();
    return DeviceStatus.connecting;
  }

  @override
  Future<bool> onDisconnect() async => await manager?.disconnect() ?? false;
}

Once the eSense device manager is in place, the eSense probe is a straightforward stream probe.

/// Collects eSense button pressed events. It generates an [ESenseButtonDatum]
/// every time the button is pressed or released.
class ESenseButtonProbe extends StreamProbe {
  @override
  Stream<Datum>? get stream => (deviceManager.isConnected)
      ? deviceManager.manager!.eSenseEvents
          .where((event) => event.runtimeType == ButtonEventChanged)
          .map((event) => ESenseButtonDatum(
              deviceName: deviceManager.manager!.deviceName,
              pressed: (event as ButtonEventChanged).pressed))
          .asBroadcastStream()
      : null;
}

Both the ESenseDeviceManager and the ESenseProbe makes use of the ESenseManager from the esense_flutter Flutter Plugin.

Adding a New Data Manager

CAMS comes with a set of build-in and external data managers - see Appendix C for an overview.

A new data manager can be created by implementing the DataManager interface.

/// The [DataManager] interface is used to upload [DataPoint] objects to any
/// data manager that implements this interface.
abstract class DataManager {
  /// The deployment using this data manager
  MasterDeviceDeployment get deployment;

  /// The ID of the study deployment that this manager is handling.
  String get studyDeploymentId;

  /// The type of this data manager as enumerated in [DataEndPointType].
  String get type;

  /// Initialize the data manager by specifying the study [deployment], the
  /// [dataEndPoint], and the stream of [data] events to handle.
  Future initialize(
    MasterDeviceDeployment deployment,
    DataEndPoint dataEndPoint,
    Stream<DataPoint> data,
  );

  /// Close the data manager (e.g. closing connections).
  Future close();

  /// Stream of data manager events.
  Stream<DataManagerEvent> get events;

  /// On each data event from the data stream, the [onDataPoint] handler is called.
  void onDataPoint(DataPoint dataPoint);

  /// When the data stream closes, the [onDone] handler is called.
  void onDone();

  /// When an error event is send on the stream, the [onError] handler is called.
  void onError(error);
}

The methods of how to handle data in the onDataPoint(DataPoint dataPoint) method as well as errors and closing event has to be implemented.

The AbstractDataManager class provides a useful class to start from. For example, the ConsoleDataManager provides a very simple example of a data manager that just prints the data.

/// A very simple data manager that just "uploads" the data to the
/// console (i.e., prints it). Used mainly for testing and debugging purposes.
class ConsoleDataManager extends AbstractDataManager {
  @override
  String get type => DataEndPointTypes.PRINT;

  @override
  Future<void> onDataPoint(DataPoint dataPoint) async =>
      print(jsonEncode(dataPoint));

  @override
  Future<void> onDone() async {}

  @override
  Future<void> onError(error) async => print('ERROR >> $error');

  @override
  String toString() => 'JSON Print Data Manager';
}

When creating a new data manager, a data manager type must be defined (it can be any string). This type is used when specifying the DataEndPoint type in the study protocol. In order to lookup the right data manager based on the data endpoint type, the data manager needs to be registered in the DataManagerRegistry.

    DataManagerRegistry().register(ConsoleDataManager());

This should happen at app start-up and before the protocol is loaded and sensing is started.

Adding Data and Privacy Transformers

Creating Data Transformer Schemas

Data transformation is supported by the DatumTransformerSchema class and can be implemented by implementing the namespace getter and the onRegister() callback function. As an example, the implementation of the Open mHealth transformer schema is shown below:

/// A default [DatumTransformerSchema] for Open mHealth (OMH) transformers
class OMHTransformerSchema extends DatumTransformerSchema {
  String get namespace => NameSpace.OMH;
  void onRegister() {}
}

Each transformer schema must be registered in the TransformerSchemaRegistry (which is a singleton). Hence, add the following line to you setup up part of the app:

TransformerSchemaRegistry().register(OMHTransformerSchema());

Once the schema is registered, transformers for each data type can be created. Data transformation is a transformation of one type of datum to another type of datum. I.e. data transformation is defined by the DatumTransformer typedef.

typedef DatumTransformer = Datum Function(Datum);

For each Datum you want to transform, you need to define a new class that extends Datum and implements TransformedDatum. For example the following OMHGeopositionDatum class represents a OMH Geoposition.

import 'package:openmhealth_schemas/openmhealth_schemas.dart' as omh;

class OMHGeopositionDatum extends Datum implements TransformedDatum {
  static const DataFormat DATA_FORMAT = DataFormat(
      omh.SchemaSupport.OMH_NAMESPACE, omh.SchemaSupport.GEOPOSITION);

  DataFormat get format => DATA_FORMAT;

  omh.Geoposition geoposition;

  OMHGeopositionDatum(this.geoposition);

  factory OMHGeopositionDatum.fromLocationDatum(LocationDatum location) =>
      OMHGeopositionDatum(omh.Geoposition(
          latitude: omh.PlaneAngleUnitValue(
              unit: omh.PlaneAngleUnit.DEGREE_OF_ARC, value: location.latitude),
          longitude: omh.PlaneAngleUnitValue(
              unit: omh.PlaneAngleUnit.DEGREE_OF_ARC,
              value: location.longitude),
          positioningSystem: omh.PositioningSystem.GPS));

  factory OMHGeopositionDatum.fromJson(Map<String, dynamic> json) =>
      OMHGeopositionDatum(omh.Geoposition.fromJson(json));

  Map<String, dynamic> toJson() => geoposition.toJson();

  static DatumTransformer get transformer => ((datum) =>
      OMHGeopositionDatum.fromLocationDatum(datum as LocationDatum));
}

The most important functions to implements is the transformer getter and the toJson() method. The transfomer returns a function that can transform one type of Datum to this type of datum. The mapping between the two types of datums happens in the fromLocationDatum factory. The toJson method is needed in order to serialize and store the data. Also note that the namespace of the DataFormat of this OMHGeopositionDatum is OMH.

Once the mapper Datum class is created, it must be added to the schema. This is done by calling;

    TransformerSchemaRegistry()
        .lookup(NameSpace.OMH)!
        .add(LOCATION, OMHGeopositionDatum.transformer);

In the carp_context_package, this is done in the onRegister() method in the ContextSamplingPackage.

Using Data Transformer Schemas

Transformation of data can be done "manually" by looking up a specific transformer schema and apply its transformer. For example, the following code transform a LocationDatum into a OMHGeopositionDatum.

    LocationDatum loc = LocationDatum()
      ..longitude = 12.23342
      ..latitude = 3.34224;
    OMHGeopositionDatum geo = TransformerSchemaRegistry().lookup(NameSpace.OMH).transform(loc);

However, a more general and common use of transformers is to specify what data format the DataEndPoint should use. This is done via the dataFormat property, which specified the namespace, like NameSpace.OMH. The data endpoint can then be added to the study runtime, when started:

  await controller.configure(
    dataEndPoint: FileDataEndPoint(
      bufferSize: 50 * 1000,
      dataFormat: NameSpace.OMH,
    ),
    privacySchemaName: PrivacySchema.DEFAULT,
  );

It is also possible to specify the data endpoint and its format as part of the study protocol:

  SmartphoneStudyProtocol protocol = SmartphoneStudyProtocol(
    ownerId: 'user@dtu.dk',
    name: 'Track patient movement',
    dataEndPoint: FileDataEndPoint(
      bufferSize: 50 * 1000,
      dataFormat: NameSpace.OMH,
    ),
  );

Creating Data Privacy

A special instance of a data transformer schema is the PrivacySchema which basically takes a datum and privacy protects relevant data properties. CARP Mobile Sensing comes with a built-in schema with a transformer namespace specified in PrivacySchema.DEFAULT. Hence, in order to add privacy protection, two things has to be implemented; (i) a transformer function that knows how to transform a datum so that data is privacy protected, and (ii) register this funtion in the TransformerSchemaRegistry.

A function that anonymize a text message datum may look like this:

part of communication;

/// A [TextMessage] anonymizer function. Anonymizes:
///  - address
///  - body
TextMessage text_message_anoymizer(TextMessage msg) {
  msg.address = sha1.convert(utf8.encode(msg.address)).toString();
  msg.body = sha1.convert(utf8.encode(msg.body)).toString();

  return msg;
}

This function can be put anywhere in the sampling package, but a typical place to to put it is in a separate file (if you have many functions) or in the package dart file (if you only have a few).

Registering of the anonymizer function is done with this statement:

TransformerSchemaRegistry().lookup(PrivacySchema.DEFAULT).add(TEXT_MESSAGE, text_message_datum_anoymizer);

This is done as part of the onRegister() callback in the package, which would look like this:

void onRegister() {
  FromJsonFactory().registerFromJsonFunction("PhoneLogMeasure", PhoneLogMeasure.fromJsonFunction);
  FromJsonFactory().registerFromJsonFunction("CalendarMeasure", CalendarMeasure.fromJsonFunction);

  TransformerSchemaRegistry().lookup(PrivacySchema.DEFAULT).add(TEXT_MESSAGE, text_message_datum_anoymizer);
  TransformerSchemaRegistry().lookup(PrivacySchema.DEFAULT).add(TEXT_MESSAGE_LOG, text_message_log_anoymizer);
  TransformerSchemaRegistry().lookup(PrivacySchema.DEFAULT).add(PHONE_LOG, phone_log_anoymizer);
  TransformerSchemaRegistry().lookup(PrivacySchema.DEFAULT).add(CALENDAR, calendar_anoymizer);
}

The example above is implemented as part of the carp_communication_package package. It is good practice to consider if your data sampling packages needs to supply privacy protecting functions, and if so, add these to the default privacy schema.

Clone this wiki locally