Yann’s Blog

I took some time to learn a bit about computer vision, using OpenCV. The end goal here is to produce a Rock Band playing “robot”, using nothing but a specially modified controller (a AT90USB + bunch of FETs) and a computer with a video capture input.

Below is a video of my first attempt at using OpenCV. It applies several filters to isolate the keys on the racetrack approximately 2 frames ahead of the goal position. The primary heuristic right now is a threshold detect filter after all of the isolation steps. If you watch closely, you can see missed or over-pressed keys, especially when the whole racetrack lights up.

Note that the video output has an overlay color square which announces the software has detected a keypress. The actual gameplay is from another YouTube video – there is no closed loop control yet.

This will take some more tuning to iron out differences, possibly preserving some state from frame to frame. Right now color information is not used, but a separate classifier pipeline and some scoring mechanic might produce better results.

I will publish the git repository with the very-alpha implementation soon.