Center for Advanced Sound Technologies, Yamaha

Corporation

VOCALOIDCommercial singing synthesizer based on sample

concatenation

Hideki Kenmochi, Hayato OhshitaCenter for Advanced Sound Technologies, Yamaha Corporation, Japan

Center for Advanced Sound Technologies, Yamaha

Corporation

System Diagram

(A) Score Editor

SynthesisScore

ConcatenationSample

Selection

Lyrics Note

(B) SingerLibrary

SynthesisOutput

(C) Synthesis Engine

•Diphones•sustained portions(Multiple pitches)

•Timing adjustment•Pitch transposition•Timbre manipulation

Center for Advanced Sound Technologies, Yamaha

Corporation

Synthesis Score

# s s I I I N N @ @ @ s s Q Q Q N N #Phonetictrack

“Sing a song” [s I N @ s Q N]

Note ON[s I N]

Note ON [@]

Note ON[s Q N]

Pitchtrack

s I N @ s Q Ns I N @ s Q N

Center for Advanced Sound Technologies, Yamaha

Corporation

Concatenation

s I I I N

Timbre(Spectralenvelope)

Useas it is

Last Frame First Frame

Fluctuations from sustained portion added

Interpolation

LocalSpectrum

Center for Advanced Sound Technologies, Yamaha

Corporation

Pitch transposition & timbre manipulation

Waveform FFT

Peak MarkingSample Pitch

(Pre-analyzed)

Pitch conversion rate

ScalingIFFT&

Windowing＆Overlapping

AmplitudeModification

Target Pitch(from Score)

SynthesisOutput

STFT

STFT

SpectralEnvelope

Center for Advanced Sound Technologies, Yamaha

Corporation

Pitch transposition & timbre manipulation

Amp

freq

Spectral Envelope

H0 H1 H2 H3 H4 H5 H6

Amp H0 H1 H2 H4 H5 H6

freq

Center for Advanced Sound Technologies, Yamaha

Corporation

Score Editor

Dedicated environment for singing synthesis Note / Lyric input Vibrato setting Control over synthesis parameters

Center for Advanced Sound Technologies, Yamaha

Corporation

Piano-roll editor

Control Track

Center for Advanced Sound Technologies, Yamaha

Corporation

Center for Advanced Sound Technologies, Yamaha

Corporation

Center for Advanced Sound Technologies, Yamaha

Corporation

Center for Advanced Sound Technologies, Yamaha

Corporation

Center for Advanced Sound Technologies, Yamaha

Corporation

Center for Advanced Sound Technologies, Yamaha

Corporation

Center for Advanced Sound Technologies, Yamaha

Corporation

Center for Advanced Sound Technologies, Yamaha

Corporation

Realtime keyboard performance mode

Lyrics input in advance Keyboard triggers note

[Video Clip]

Center for Advanced Sound Technologies, Yamaha

Corporation

VOCALOID Products

Version 1 (released in 2004)

Version 2 (released in 2007)

LEON LOLA MIRIAM Meiko Kaito

Big-AL

Sweet ANN

Prima Miku

Center for Advanced Sound Technologies, Yamaha

Corporation

Audio Demos

“Amazing Grace” Compulsory

Center for Advanced Sound Technologies, Yamaha

Corporation

Acknowledgements

The basic signal processing technique used in Vocaloid is developed though a joint research project between Yamaha Corporation and Music Technology Group (MTG), Universitari Pompeu Fabra, Barcelona.

The authors would like to thank the staff at MTG, especially to Mr. Jordi Bonada and Dr. Alex Loscos, for their contribution to Vocaloid.

Center for Advanced Sound Technologies, Yamaha

Corporation

Dank voor uw aandacht.