SoundScopePad

RIKEN’s SoundScopePad provides an active music listening experience using AI, virtual reality (VR), and spatial acoustics. Users can find the performer that they want to hear and focus on them by emphasizing their performance.



- VR concert experience
Performers stand around you and start playing, and you can see three at a time in front of you on your iPad screen. If you point your iPad to the left, the performer on the left will appear, and if you point it to the right, the performer on the right will appear.



- Spatial acoustic experience
If you hold your iPad in front of you, you will see the application tracking your face on the display. By turning your head left and right, you can follow the sound of the performers on their respective sides. Likewise, if you turn around with your iPad in hand, you will be able to face the performers behind you.



- Active music listening experience
This app provides an active music listening experience by enabling you to focus on certain performers that you want to hear. If you turn to face the performer of your choice, it will sound as if they are directly in front of you. By moving your left or right palm closer or further from your ear, you can adjust the range of the performance that you want to focus on. You can highlight one part or multiple parts of the performance such as the horn section and the rhythm section. You can check the direction of the performer and the direction you are facing on the position display.



- Development history
The Music Information Intelligence Team, part of the RIKEN Center for Advanced Intelligence Project, is exploring potential new ways of creating and listening to music using AI. When listening to classical music, for example, you may want to hear the first violin a little more clearly, or suppress the sound of the trombone sound, or only hear the flute. SoundScopePad is an app that enables you to specify the sections you want to listen to.

- Data Collection and Use
The MusicScopePad does not collect personally identifiable information.
The handling of the face image data acquired by the front camera is specifically explained below.
Collection：
Face image data is not saved.
Use：
The face image data is used only for grasping the angle of the head.
Disclosure：
Face image data is not saved and there is no information to disclose.
Sharing：
The face image data is not saved and is not shared.
Retention of face data：
The face image data is not saved and is not retained.


The app is available to the public to gauge the overall interest in such a way of listening to music. We ask that users the app who are prompted to fill out a questionnaire to please submit their responses. We plan to add songs in the future. If you are interested in the app and technology, please contact us at the support URL.