SoundTracer

Discover traditional Norwegian music by drawing gestures with your iPhone or iPad.

First draw a gesture by moving your iPhone or iPad up and down, to indicate ascending and descending melody. A simple sound is played while you move your device, its pitch gets higher when you move up, and gets lower when you move down. You can also indicate the beginning of a new note by moving abruptly the phone up or down (a blue vertical line indicates the note position). It is also possible to indicate a particular note without indication of pitch by performing a quick movement down and up (a red vertical line indicates the note position).

Once you have terminated your gesture (by tapping once), the piece of music that contains the melodic gesture closest to your gesture is presented. The title and genre of that piece of music is indicated, as well as the performer and the district in Norway where it comes from. A melodic curve of the specific melodic gesture from the song is displayed below your own gesture. The matching between your gesture and the melody is based on a temporal alignment of their contours: the ascending parts of your gesture is aligned with the ascending parts of the melody, and same for the descending parts. The temporal note locations you have specified (bleu and red vertical lines) are also tentatively aligned with corresponding note changes in the melody. Some parts may be skipped, and are shown in red in the curves. When the music is played, a cursor moves throughout both curves so that you can follow the music while looking at the gestures.

SoundTracer uses Apple's ARKit technology, solely in order to precisely track the location of your iPhone or iPad. For that reason, there is no actual Augmented Reality, and the camera is solely used to improve the precision of the tracking, without displaying the actual frames of the camera.

For the moment, the catalogue of music is composed of around 50 pieces, and only the initial part of the melody is considered. The catalogue includes a few tunes played on the traditional Norwegian Hardanger fiddle, and songs sung a cappella. SoundTracer will be progressively improved with more music.

SoundTracer is an innovation project led by Alexander Refsum Jensenius, supported by the University of Oslo and developed in collaboration with the National Library of Norway. More information here: https://www.hf.uio.no/ritmo/english/projects/all/soundtracer/