I recently had time to tinker with my NeuroSky MindWave - a cheap passive consumer EEG. I’ve decided to build a simple menu controlled by deducing blink events from the raw data stream the device offers to developers. All code discussed here is available in a GitHub repository.
For now the device is connected to a PC using the supplied radio receiver that functions as an A/D converter and supplies a virtual serial port.
For more information on the device itself, check out the product page or the wikipedia article.
Though NeuroSky offers quite a few abstraction layers on top of the serial data stream, I’ve decided to take the direct approach by communicating with the device over a serial connection. The motivation for this was that I eventually want to do all of this processing on a micro processor anyway. The API reference is available as part of the MDT at the NeuroSky wiki. It provides detailed explanations of the serial data stream and good guidelines for implementing a parser.
The parser is written in Java as part of the Processing sketch at https://github.com/FrankGrimm/readmind/tree/master/processing/readmind. It uses the processing serial library to communicate with the receiver and processes all incoming data in a seperate thread. This was necessary because the frequency of incoming input data (for the raw signals) is much higher than the number of drawings per second. In order to perform statistical analysis in near real-time the resolution of the current state at the time of drawing the visualizations simply wasn’t sufficient. Whenever a valid packet payload is decoded, the background thread notifies different components of the incoming new datapoints. These components are primarily data buffers for the visualization and several classes that perform live evaluations on the signal data for detecting blink events.
While implementing the parser (and fixing bugs… and fixing bugs…) I’ve hit quite a few pitfalls:
Most applications for this device rely on attention and meditation levels. While these values are calculated by an proprietary, undisclosed, algorithm they could have come in handy when interpreting the raw data points. Especially the attention level seems to correlate highly with the reliability of blink events. While the implementation guide specifies signal codes for attention and meditation levels, as well as blink event signals, none of those packet types are sent by the device I own. In case you need these values the best way to obtain them would be to use one of the other high-level APIs or play man-in-the-middle between the official socket wrapper and the serial device. The approach discussed in the next section provides “good enough” results for detecting blink events though.
The signal quality, especially of the raw signal, depends highly on a firm seating and minimal facial movement during the usage of the device. If there’s too much activity, the reference wave and the measure of the EEG sensor will differ, resulting in larger ampltitudes of the raw signal and unclear readings. While the protocol offers signal quality packets, these are only an indication of the connection between the radio transmitter and the USB dongle. This behaviour results in a lot of false positives, which are an open issue on my todo-list for this implementation. The only way to deal with those rapid changes in amplitudes are to ignore rapid successive blink events, which leads to false negatives when the user simply blinks fast enough.
The low resolution of the signal values for the 8 EEG bands makes the data pretty inconclusive for any useful live interpretation from my point of view. I’ll have to figure out if I can do some kind of attention / meditation aggregation on those values myself at some point.
Detecting the device
I dislike fixed device IDs so I wrote a rudimentary synchronous version of the parser that enumerates all available serial ports and tries to read a full packet of valid (as in valid checksum) MindWave data. If no such packet is received in the first few hundred bytes, or a timeout is hit, the device id is discarded and the code moves on to the next serial port. The implementation for this is contained in the class MindwaveDeviceFinder in mindwave_find.pde.
The parser thread continuously updates several buffers, one large buffer for 10240 raw values and smaller buffers for each of the 8 supported EEG bands. The bands are not updated often so smaller buffers are sufficient. The implementation for this ValueBoxDrawer class holds and draws the buffered data. It is located in readmind.pde. I implemented a branch where all of the band graphs share a common y-axis scale but this basically lead to some bands not being readable at all so I’ve discarded that approach. For now, all the graphs y-axis are scaled to showing MAX(buffer)-MIN(buffer) of values. The x-axis depend on the size of the buffer that’s plotted by the instance of ValueBoxDrawer (all band buffers have the same size, the raw buffer is much larger).
The last read signal quality packet (0-255) is displayed in a little bar graph, similar to the ones seen on mobile phones. The value is not buffered because it is not really used for anything other than user information at the moment.
The major portion of the screen is used by a simple rectangular menu section (containing menu items and sub-menus). This menu can be controlled by mouse events and blinking (this is discussed later on).
The repository contains a few R scripts (get R here) I’ve used for visualizing and analyzing the raw signal and band data.
Both enumerate all the exported raw- and band-data files in the working directory and plot the results of several calculations in the same directory.
This script exports a visualization of the raw signal data together with some simple moving averages that have been used to estimate factors in the processing implementation (see section “Blink detection”).
This script exports visualizations of all the raw signal data, as well as the bands in three different files (raw-only, color coded bands-only and a combined image).
// works for my brain
For this first iteration, a simple moving average is calculated over the last Navg (=400) raw values. Everytime a new raw value is received, the oldest raw value in the queue is subtracted from a cumulative sum and the new raw value is added. By dividing this sum by the size of the buffer, the unweighted average over the full size of the buffer is calculated. The average value has the same dimension as the raw values and can be used to do simple comparisons between the current average and the current raw value.
The average value is then multiplied by a static factor. All peaks in amplitude above this value are considered part of a blink event.
I’ve used the R script (graph-means.r in the repository) to tinker with the size of the buffer (Navg) (the cardinality of the subset of all raw data received at a given time that’s factored into the moving average), as well as the scalar factor (= 2.5) used to separate regular noise / signal movements from peaks that could potentially be associated to blinking.
The Java implementation for the same analysis on live data is located in the blinkdetect.pde source file. Using these peak amplitudes as they occur would result in a lot of false positives and a large number of short, duplicate events that would normally be attributed to the same blink. To combine closely related peaks and discard outlying values that don’t appear to be related to other peaks I’m using an instance of BitSet (with a fixed size of Nblink (with Nblink < Navg, = 100)). Whenever this smaller buffer accumulates enough peaks (>= 30) a blink is considered to be started and the current time (in milliseconds) is recorded. If a blink event is started and the number of peaks in this buffer drops below that same value, the blink is considered ending. In this case, the duration is calculated and a blink event is triggered in the main class (by invoking the blinkHandler method).
Controlling the menu
In order to deal with the high number of false positive detections of blink events I’ve introduced a timer to activate menu items and ignore certain blink events. The currently set duration of the timer represents the state of the current activation of a menu item.
- Regular state: No menu item is selected, the menu succeeds to the next item in the list every 700ms. This succession is only applied in the regular state.
- Initial state: A blink event occured. To prevent large numbers of false positives (and to prevent ending up in a totally different menu) the timer is set to 500ms, ignoring all other blink events during this time.
To deal with rapidly occuring false positives, one could add a counter to the initial state and abort the selection process if too many events occured during this timespan.
- Selected state: The initial state elapsed without cancelling the selection process. The timer is then set to 3000ms and the currently selected menu item is highlighted with a red border.
If there is another blink event during this timespan, the menu item is activated (meaning an action is triggered or a sub-menu is entered). Otherwise the timer is set back to 700ms, which places the system back into the regular state and will advance the menu to the next item.
There’s still a little TODO marker in the drawing section for displaying the keyboard in a grid instead of a (much too large) list like the other menus.
Other than that, I’ll need to figure out a way to limit false-positive recognitions of blink events on the software side of things in order to get this to a state where the menu inputs are fast and reliable enough to control text input or movements for one of my robots. At the current rate and with all the false-positives the only thing that would stop a robot controlled by this setup from bumping into walls all the time are the ultrasonic sensors I’ve attached.
Another goal is to turn this into a USB-HID and getting rid off the huge delays for simulating keypresses or menu actions. The FFT is done in the headset component and the receiver side offers RX/TX lines for serial communications with a development board. I didn’t yet find specific schematics for the receiver though and I don’t really want to break it in the process (apparently some solder lines need to be cut to get a clear serial communication going).
Like this post? Follow me on Twitter or get in touch!