We built this computer vision system that can play Rockband as our final project for CS184 at UC Berkeley.
In the space of two weekends we designed and built a system that uses computer vision to monitor the xbox display through a camera and play the game. More details can be found at inst.eecs.berkeley.edu/~njoubert/cs184/Rockband.pdf
We were, well, sleep deprived for a good section of this work, which explains the craziness in some parts of the video.
Our system is similar to Slashbot and AutoGuitarHero, but we do not take a video feed from the console – no, we’re doing it through a panasonic handycam pointed at the screen. We’re interested in machine vision, and this was a fun project to get into the field!