4 Degrees of Separation

An ML-powered icebreaker inspired by camera apertures

What:

Physical Computing, Interaction Design and Machine Learning

Who:

Imperial College London

When:

2024

Context:

My classmates and I were interested in how machine learning and mechanical motion could be combined to create a playful, interactive ice-breaker for two strangers.

Outcome:

A game with 4 motorised apertures and 2 webcams, with two players facing opposite ends. A regression-based ML model tracks their hand gestures, gradually opening the apertures in proportion to how similar their hands are relative to a hidden “key” gesture. Guess the key, and proceed to the next row of apertures. Guess the next key, and the apertures snap open, revealing each other's faces.

My Role:

I was the programmer for this project, where I created a pipeline using C++, Processing, Serial and OSC. Much of my development was focused on connecting two Arduinos, each receiving inputs from separate regression models tracking the players’ hand gestures. By connecting them, the apertures would only open if both players simultaneously held up the correct key gesture.

For our game, we used Handpose OSC to process input data from two webcams. This system leverages a pre-trained model to precisely estimate the 3D position of the player’s hand using 21 hand keypoints, resulting in 63 data points as input for our neural network, which was built with Wekinator.

We trained a regression model to distinguish between two hand gestures, one for each level in the game. The output of the model was two float values between 0 and 1, which represented an estimate of how close the player’s gesture was to the secret ‘key’ gesture. A second identical model was run in parallel, sending appropriate values for the other player.

The output data from the model was then sent via two serial ports to an Arduino, which turned the servo motors proportionally to the input values. If these inputs were equal to 1 for both players simultaneously, the level was passed and the apertures fully opened.

To build the physical apertures, we used MDF, acrylic, screws and 3D printed spacers and gears. We greatly reduced the friction in our design by controlling the movement of the aperture wings with a large gear ring, which doubled as the aperture window.