We have proposed; as a long-term goal, the development of musical performance robots designed to be able of interacting with musical partners (both human players as well as musical performance robots) at the emotional level of perception.
For this purpose, we are focusing our research in enhancing the perceptual capabilities of the anthropomorphic flutist robot as well as developing a new musical performance robot.
In this research, we are focusing on developing a new musical performance robot which in the future will be able of interacting with the anthropomorphic flutist robot. For this purpose, we have decided to develop a musical performance robot which is able of performing other kind of wind instrument.
Wind instruments fall into one of the following categories: brass instruments and woodwind instruments. One important difference between woodwind and brass instruments is that woodwind instruments are no-directional. This means that the sound produced propagates in all directions with approximately equal volume.
Brass instruments, on the other hand, are highly directional, with most of the sound produced traveling straight outward from the bell. Thus, the wood instruments represent a higher challenge to reproduce the human motor control from an engineering point of view. Basically, there are three types of wood instruments: single reed (i.e. clarinet, saxophone), double reed (i.e. oboe, bagpipes, etc.) and the flute. In particular, we have focused our research on developing an anthropomorphic saxophonist robot since 2008.
Anthropomorphic Saxophonist Robot
WAS-4R
WAseda Saxophonist No.4 Refined
Objective
Overview
In 2015, we have developed the WAseda Saxophonist Robot 4 Refined (WAS-4R) which has been designed to improve articulation by the combination of airflow control and tonguing.
In particular, the WAS-4R is composed by 32-DOFs that reproduce the physiology and anatomy of the organs involved during the saxophone playing as follows.
| Part | DOF |
|---|---|
| Lips | 2 |
| Tongue | 1 |
| Oral Cavity | 1 |
| Lung | 2 |
| Fingers | 19 |
| Eyes | 2 |
| Waist | 1 |
| Eyebrow | 1 |
| Eyelid | 2 |
| Total | 31 |
Lips
Lips
Lips Mechanism
The lip mechanism consists of 2-DOFs designed to control the up/down motion of both lower (pitch changes are controlled) and upper lips (sound pressure changes are controlled). In addition, a passive 1-DOF has been implemented to modify the shape of the side-way lips.
Oral Cavity
Oral Cavity
Oral Cavity Mechanizm
The oral cavity mechanism is consist of 1-DOF designed to control the up/down motion of sectional area of oral cavity. It is able to control sound pressure by the motion. This mechanism is designed by mesureing proffesional music performer.
Tongue
Tongue
Tongue Mechanism
The tongue and oral cavity are made of SEPTON. As shown a left figure, the tongue is in contact with the reed of saxophone and the applied torque is transferred from motor to tongue by link mechanism. As a result, robot can control the dynamic properties of the sound such as "attack" and "release". In addition, in order to increase the range of sound, the shape of the oral cavity has been re-designed based on the measurements obtained from ultrasound images taken from professional saxophonist players.
Lung
Lung (Air compressor)
Lung Mechanism
Regarding the air source (lung mechanism), a DC motor has been used to control the motion of the diaphragm of the air pump; which is connected to an eccentric crank mechanism. This mechanism has been designed to be light and small. And two valves are used for the air flow control. One is the needle valve that can rectify the air flow from the pump.Another valve is a solenoid valve that can control the air flow rapidly.
Fingers
Left Hand Fingers
Right Hand Fingers
The alto saxophone is possible to play from A#2 to F#5. For this purpose, a human-like hand has been designed, which it is composed by 19-DOFs.
In addition, in order to compensate the delay on the response of each finger mechanism, rotary encoders have been embedded to compensate the dead time. As a result, the response of all fingers can be synchronized during the performance.
Eyes
Eyes
Eyes Mechanism
We added an eye mechanism to enable the robot to perceive other musicians in its environment and to react to their performance actions. The mechanism contains two CCD cameras that can be moved in 2 DOF (pitch and yaw axis). The mechanical capabilities (velocity, viewing angle) of the mechanism are designed to resemble the characteristics of the human vision apparatus.
Waist
Waist
Waist Mechanism
The waist mechanism is consist of pitch DOF designed to show physical cue like a musician. It is implemented Mechanical Gravity Canceller and operates smoothly.
Facial Expression
Facial Expression
Facial Expression Mechanism
Facial expression mechanism is consist of eyebrow part and eyelid part, and it eneable more expressive perfomance. About the eyebrow part, raising eyebrows and frowning are shown with 1 DOF by using a round belt. About the eyelid part, it has 2 DOFs and opening and closing eyelids are shown by using a timing belt.
Control System
This block diagram is the WAS-4's control system. In the fingers, rotary encoders measure every finger's delay time. Additionally, in the pressure-pitch controller with Finger synchronization, they are controlled by FF control system. FF control system is constructed by Neural Network's study from FB value. In the system, pressure, note, lip position and pitch are inputs. And valve position and lower lip position are outputs. Also Fluctuation Generator generates 1/f fluctuation that can performe more human like sound.
Performance System
The MIDI accompaniment system of WAS-4 consists of two computers to control the robot's musical performance: one for controlling the robot, and the other to generate the accompaniment MIDI data. These computers are connected by the MIDI system, and the synchronization of the performance is achieved by using the MIDI signal.
Interaction System
WAS-4 enables to synchronize with human saxophonist the timing of the start and timing of the end of performance. During the music performance, human saxophonist has constant movement. So WAS-4 detects motion information by IMU sensor and CCD camera on eyes and tries to match the timing of performance.
In performance with dancer, WAS-4 detects motion information by IMU sensor and CCD camera and change its playing phrase.
Performance Movie
Special Thanks
A part of this research was conducted at the Humanoid Robotics Institute (HRI),
Waseda University.
WAS-4 has been designed by 3D CAD software "SolidWorks".
Special thanks to SolidWorks Japan K.K. for the software contribution.
Humanoid Robotics Institute, Waseda University
SolidWorks Japan K K
TOYOTA Motor Corporation