PaintOnMusic comprises a four-color drawing mechanism over users’ webcam, and three interchangeable musical sections designed and manipulated in Max and Ableton Live. The drawing mechanism is built with Python through the OpenCV library. It tracks an object’s movement, with which users can paint on top of their webcam by moving the object around. The drawing is constantly analyzed with Python, and the result is continuously being ported into Max. Such analysis controls in real-time several sound parameters, such as gain, panning, and delay, which are designed with Ableton Live.
Our system can be used as a pedagogical tool for sonification, or simply as a fun way to simultaneously generate live visuals and live audio. Nonetheless, for our system to reach its full potential, we encourage our users to explore the extremes graphically and musically: paint with all colors to make layered voices, paint left and right to pan different sounds, paint up and down to drive the effects, etc. After all, crossing perceptions, like those of sight and sound, is the key to new discoveries.