Week 9 - Device/Data connection | The four F's of active reviewing

Facts:

After completing most of the visual components I plan to start this week on how to integrate the EEG device I selected, the Unicorn Hybrid Black, as a medium to connect users with the program.

The goal of this stage is to enable users to experience the Unicorn Hybrid device, using real-time EEG data (raw) detected by the device, encoded via Python scripts, then collected and stored in Unity through the Lab Streaming Layer (LSL). [The lab streaming layer (LSL) is a system for the unified collection of measurement time series in research experiments]. Once the usable data is present in Unity, I can create C# scripts to control the EEG frequencies, using detection and recognition to implement the commands I need, which in this case, is the spacebar.

• Before officially proceeding with this stage, I had a one-and-a-half-hour online meeting with Kunal to resolve issues with device connectivity. We later discovered that the reason the Unicorn Hybrid could not connect properly to my Windows computer was a conflict between the device's Bluetooth and the computer's built-in Bluetooth. This means that, in the future, I will need to disable the original Bluetooth when I want to connect the device and not connect other Bluetooth devices like a mouse or headphones.

• During the online meeting, we also encountered a significant issue with installing the built-in LSL in Unity on my Windows computer. We tried multiple times without success, and Kunal then suggested exporting my project as a package and importing it into the 'Angry Bird' project (which has LSL built-in).

• I scheduled another appointment with Kunal on Thursday morning to help resolve the technical issue. After we reviewed the LSL problem, we found that the overall performance was poor, and it was not functioning properly. He then recommended an alternative approach called Open Sound Control (OSC), a protocol for networking sound synthesisers, computers, and other multimedia devices for purposes such as musical performance or show control.

• I downloaded the required OSC package from GitHub (website) and imported it into Unity's Assets for operation. Kunal then helped me update the OSC-Python script. I had initially attempted to modify the Python script to import the extracted raw frequencies into Unity, but after failing, Kunal generously helped me solve this issue.

After all these steps, when the user wears the Unicorn Hybrid device and opens LSL, they can then run the software (Visual Studio Code / Microsoft Code, etc.), execute eeg_process.py, and see the user's raw EEG data in the terminal.

Feelings:

I have been feeling very anxious during this step, as it involves coding and computer operations that I'm not skilled at. Even though I took a conjoint computer science course in my first year, my basic Python background isn't sufficient to complete such a complex project. Fortunately, I found out that Kunal had previously worked on a similar project called 'Angry Bird,' which also used brainwaves to play a game, controlling the bird's flight path and strength to defeat the 'green pigs.' He mentioned that I could borrow the Python scripts from 'Angry Bird' to extract the raw data.

Findings:

After multiple trials, I found that the data connection throughout the process is very unstable, with frequent errors appearing in the terminal, such as data extraction failure or inability to connect to the device. These issues are often caused by factors such as the device running out of battery, the device not being connected, LSL connection failure (requiring a restart), or the need to restart the Python script.

Future:

The next step, with Kunal's guidance, will involve using the raw data to extract the user's focused brainwaves and baseline brainwaves separately in Unity. The former will be accomplished through my Tutorial scene, where the user's focus values will be extracted and saved to trigger the spacebar command. In the later stages, whenever the user's focus value meets the threshold, interactions will occur to achieve the effect of controlling the screen's operation using brain activity.