Air Piano: Playing Piano Without a Piano Using MediaPipe and Unity

Publication Date

Spring 2026

Presentation Length

Poster/Gallery presentation

College

College of Sciences & Mathematics

Department

Math and Computer Science, Department of

Student Level

Undergraduate

Faculty Mentor

Christina Davis

Metadata/Fulltext

Fulltext

Presentation Type

Gallery

Summary

Air Piano explores Google’s MediaPipe Hands model, using a MediaPipe Unity plugin developed by homuler to bring real-time hand tracking into Unity. I take these inputs and map them onto a hand model modified from the open-source Ultraleap hand tracking project, allowing tracked hand data to interact with the virtual environment.

A major part of this work involved converting MediaPipe XYZ hand landmarks into joint rotations contained in a Unity skinned mesh, requiring coordinate transformations and handling noisy input to maintain stable, real-time interaction. These calculations are used to project hand movement into a constrained piano interaction.

The result is an interactive “air piano” where users can play notes using just their hands and webcam, no physical instrument needed.

This document is currently not available here.

Share

COinS