Neurostack + MindType

Writing neurotech apps can be really fun. It can also be quite daunting. A huge reason for that has been because it's super-hard to write your own ML algorithm, figure out how it'll communicate in real-time, and deploy it to the cloud. So with Neurostack, we decided to do that for you!

Neurostack is an open-source cloud EEG service that you can use to rapidly prototype your own neurotech application. The Neurostack client (P300Client) accepts a stream of MuseLSL data, and forwards it to the appropriate service. The service then analyzes the EEG and returns a stream of target vs non-target epochs. So far, Neurostack has the following services:

• P300Service: provide samples of target and non-target data once, and P300Service will find P300 brain waves for that person
• NLP+EmojiService: provide a word / phrase, and receive word and emoji predictions

How to use

1. Start a MuseLSL stream
2. Connect to P300Client via SocketIO on your front-end
3. Send training markers (where p300 is either 0 or 1 - boolean)

emit("train", (uuid, timestamp, p300), callback_function)

4. Ask for a prediction, and get your results! Returns sid, (uuid, p300, score)

emit("predict", (uuid, timestamp), callback_function)

For more information, look at the README in: Code/src/p300_service.

MindType: A Neurostack application

Communication, especially via keyboard, is very difficult, if not impossible, for people with various neuromuscular degenerative diseases and muscular dystrophy. Because of this, EEG spellers based on the P300 oddball paradigm have been made and researched upon for many years. Current EEG spellers are quite slow (~50 bits/min, or 6 letters/min with NLP optimizations) [1] [2]. It becomes impossible for a person with such conditions to enjoy a conversation with their loved ones due to the low bit rate.

MindType seeks to improve the bit rate of mind-controlled keyboards. It also uses a grid system, like traditional P300 spellers. We have included several other features along with this, such as an emoji keyboard, for more expressive typing in the modern world. Additionally, the use of NLP algorithms would boost the bit rate by attempting to predict the word the user wants to enter. This allows for many cases where the user wouldn't have to type the full word.

We are currently wrapping up integration for MindType, so stay tuned for the beta release! Note that MindType currently only runs on Mac or Linux.

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3679217/

[2] https://www.ncbi.nlm.nih.gov/pubmed/12853169

Overview

P300 Client and Server

This is a client-side application. Running the P300 client starts up 3 streams, all of which function in real time:

1. EEG stream, which connects to the Muse headband and streams EEG data through an LSL outlet.
2. Marker stream, a stream of timestamps coming from the front end. These markers represent requests for either training or prediction for the classifier, based on the EEG data at the timestamp.
3. ML stream, which takes the relevant pieces of EEG data from the EEG stream based on the timestamps from the Marker stream and does some basic preprocessing.

The P300 client then sends machine learning jobs to the P300 Server, which trains classifiers and predicts based on the data sent. The result is sent back to the client, and then to front end. For more detailed documentation on how the interactions between client and server work, check the README.md in the p300_service folder.

Hardware necessary:

Muse, a BLED112 dongle if on Mac.

Dev Setup

Front End:

Setup:

1. npm install
2. npm install -g concurrently
3. npm install -g wait-on

To run dev Electron build:

npm run electron-dev

Backend:

To start the MuseLSL stream, follow the instructions for BlueMuse

To run the P300 client:

1. Go into ./Codr/src/p300_service
2. Install the requirements.txt with pip
3. Run python p300_client.py

This will create a P300 client, start the streams, and connect to the P300 server which we have running on GCP.

Contributing

We follow trunk-based development to avoid merge conflicts as much as possible, and to ensure that code in our master branch is always works.

1. Make a new branch for your small feature.
2. Code :D
3. Once your small feature is done, make a pull request. If your feature isn't fully ready, use feature toggling to turn it off (feature_flags/feature_flags.py)
4. Your code will go through a review. Once it passes, merge the pull reqest!
5. Repeat :D

P300 keyboard: Algorithm approach

We used a linear discriminant analysis (LDA) and vectorizer pipeline, as well as a ERP Covariances and MDM pipeline (from Alexandre Barachant's notebook), to classify P300 from the Muse. We obtained samples between 0.1--0.75s after the stimulus, and passed it to the classifier as a single vector. We achieved similar results (accuracy of roughly 60-70%) from both of these classifiers, possibly due to some noise in the data.

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3175727/
  • "Comparison of Classification Methods for P300 Brain-Computer Interface on Disabled Subjects"
• http://alexandre.barachant.org/blog/2017/02/05/P300-with-muse.html
  • Thanks to Alexandre Barachant for his P300 script on Muse! Unfortunately it was on Muse 2016, so we made a Muse 2014 version of it

Design Decisions

The Neurostack is designed with the microservice pattern. This offers two advantages. First of all, this allows us to incrementally upgrade services as we improve our machine learning capabilities. Most importantly, this lets neurotech enthusiasts (including us!) to mix and match different services for custom needs, or even create new Neurostack services out of existing ones. This would be especially useful for multimodal analysis.

We decided to opt for MuseLSL over MuseJs as the connector to Muse as it allowed for superior Bluetooth reliability and time-syncing. The Python-based Neurostack client facilitates communication between the Neurostack cloud and a front-end of your choice using SocketIO. This gives developers freedom of stack, while guaranteeing reliable streaming. Each service comes with its own database to store users and their training metadata used to predict neurological phenomena for that person.

Current concerns include security and privacy. We plan on encrypting all data in transit, and using UUIDs to identify users instead of accounts. This would allow for anonymous analysis of EEG data.