A System for Presenting Background Scenes of Karaoke Usingan Active Database System

Tsutomu TERADA Masahiko TSUKAMOTO Shojiro NISHIOCybermedia Center Grad. School of Information Grad. School of InformationOsaka University Science and Technology Science and Technology

tsutomu@cmc.osaka-u.ac.jp Osaka University Osaka Universitytuka@ist.osaka-u.ac.jp nishio@ist.osaka-u.ac.jp

Abstract

In recent years, it has become popular to use mul-timedia data for building presentations. In cases ofreal-time contents are built, a powerful mechanism isnecessary for extracting materials suited to the situa-tion from a large amount of multimedia data. In thisstudy, as an example of such a mechanism, we con-structed the Active Karaoke system, which generatesbackground scenes for karaoke using an active database.An active database is a database system that processesprescribed actions automatically in response to the oc-currence of an event. We show that Active Karaoke canextract appropriate multimedia data from the databasein response to the change of music tones and contents oflyrics, and display them as background scenes. Since anactive database is used for storing and managing data,the system behavior can be easily and flexibly describedusing ECA rules, which are the behavior descriptionlanguage of an active database.

1 Introduction

In recent years, the evolution of multimedia technolo-gies has focused a lot of attention on techniques for us-ing multimedia contents in databases for real time pre-sentations. A real time presentation is one that showsmaterials extracted in real time from a database in re-sponse to a change of situations without prescribed sce-narios (Figure 1). Since situations come under the in-fluences of a system state and external environments,a real-time presentation-generating system must ex-tract appropriate materials from databases in responseto a change of situations. In this study, we pick upkaraoke as an example of real time presentation sys-tems. Karaoke is the act of amplified singing to tapedaccompaniment. It is a popular form of entertainmentin Japan, and also gaining popularity around the world.

Although singing songs is a prime purpose of karaoke,

Real-timeExtraction

Presentation

Database

Figure 1: Real-time Presentation System

it is also a good way to communicate with other peo-ple. It is this reason that personal karaoke systemsare not widespread, despite high popularity of karaokerooms[4]. Consequently, karaoke systems have been de-veloping with various technologies to make singing ex-citing.

A karaoke system is composed of a singer’s voice,background music, a background video with lyrics andother miscellaneous elements. Many functions to ex-cite singers have been already realized on commercialkaraoke systems. For example, with respect to voices,the “virtual duet” function with only one singer plusa voice effect function makes the singer excited. In re-gards to background music (BGM), the user can singcomfortably with the automatic transforming functionto adjust the user’s voice to the exact pitch. In terms ofother functions, DAM-DDR is a function where a usercan dance while singing in accordance with instructionsdisplayed on the screen of the karaoke system. CalorieKaraoke is a function that displays calories consumedby singing, and so on[1].

On the other hand, most of back ground videos forkaraoke play nothing but a video movie that is related tothe song to some degree. Hence, the goal of our study isto develop a system for generating background scenes ofkaraoke as an example of real time presentation generat-

Music Information

Database

BGV MaterialsDatabase

Rule Base

Active Karaoke

External input

Information oflyrics and musics

Constraint from rules

grace cup, I love you～

Figure 2: The System Image of the Active Karaoke

ing systems. This system generates background scenesdynamically in response to changes of situations.

The remainder of this paper is organized as follows.Section 2 outlines the Active Karaoke. Section 3 ex-plains an active database that forms the base technologyof our system, and Section 4 illustrates an implementa-tion of Active Karaoke. Section 5 shows related works,and Section 6 presents conclusions and future works.

2 Active Karaoke

Active Karaoke generates and shows backgroundscenes of karaoke suited to the BGM. This system ex-tracts scenes from databases dynamically and automat-ically in response to changes in lyrics, melody partsand the environmental mood. The system image is il-lustrated in Figure 2. In the figure, the BGV (BackGround Video) materials are a database to store ma-terials for karaoke background scenes. Each data hassome information, i.e., key words, file size, etc. TheMusic Information Database is a database for storingmusic data, consisting of lyric data, lyric display tim-ing, music genre and others. The Rule Base has rulesfor deciding what scenes are displayed and what effectsare given.

While playing a song, Active Karaoke continues to re-trieve and display materials automatically in real timeby exploiting information stored in the Music Informa-tion Database, the external input, and rules in the RuleBase.

Active Karaoke has the following characteristics:

• Scene selection keeping up with the progressof the playing music: Active Karaoke extractsa suitable scene from the database in response tochanges in lyrics, melodies and the other environ-mental mood.

digital camera

Picture dataon today

Figure 3: Practical model 1

• Various visual effects: The system can applyvarious visual effects to displaying images for ex-citing karaoke. For example, if a music playing is aslow ballad, the system displays scenes with colortones changed to soft colors.

• Utilization of personal multimedia data:Users can easily register BGV materials such asprivate photos. We expect the system to attracta lot of interest with these personal materials asbackground scenes.

The following models are examples of practical usesof Active Karaoke:

1. Enjoying Active Karaoke with photos takenat the place where users are staying (Figure3): A user and friends go on holiday to an amuse-ment park with digital cameras and take many pic-tures there. When users enjoy Active Karaoke ina karaoke room on the way home, pictures thatware taken by the users can be used as backgroundscenes by transferring photo data to the karaokesystem. If the digital camera has a function for at-taching keywords to pictures, our system can showthe picture at applicable times in consideration ofthe attached information.

2. Enjoying Active Karaoke with prepared pic-tures on a network (Figure 4): A user pre-pares pictures with keywords, and uploads thesepictures to a WWW site in advance. The user en-joys karaoke with prepared pictures as backgroundscenes by entering the URL of these pictures at thekaraoke house. If a user prepares several sets ofpicture data on Internet, he can choose a picturetheme by changing the entered URL.

Internet

Registration of pictures on the network

Input URL fordownloading these pictures

http://www.ise.eng/mypic/*

Figure 4: Practical model 2

3 Active Database

The most important function of Active Karaoke is tocarry out various actions in response to changing sit-uations. For example, Active Karaoke has a functionfor applying effects to a scene in response to changes inlyrics, and to change scenes in response to an externalvoice input. Especially in the latter case, since an ex-ternal input is the accidental matter for the system, itis difficult to predict these events. As we mentioned inSection 1, since karaoke systems have been continuouslydeveloping, it is necessary that the system is adaptable.

Thus, we constructed Active Karaoke using activedatabase technologies. An active database is a databasethat processes the prescribed actions in response to theoccurrence of an event arising on inside/outside thedatabase[5]. Behaviors of an active database are de-scribed by ECA rules. Each ECA rule consists of threeparts: Event, Condition, and Action. The Event partdescribes the occurring event in the system. The Condi-tion part describes the conditions for executing actions,while the Action part describes the operations to becarried out.

Using ECA rules for describing the system functions,we can describe system behaviors in an event-drivenmanner. Moreover, we can customize system functionsby adding/deleting ECA rules to/from the system.

3.1 ECA Rule

In general, conventional active databases are onlycapable of detecting the events that occur within thedatabase, such as SELECT, INSERT, DELETE, andUPDATE. Correspondingly, active databases can carryout only database operations as actions. In our re-search, we enhanced the description capability of ECArules to fulfill the various requirements for karaoke sys-

� �CREATE RULE Rule name ON Event name

[ Definition of variables ]

[ WHERE Conditions ]

THEN DO Actions

� �Figure 5: Syntax of the ECA rule

Table 1: Events provided by Active KaraokeName Content

SELECT Data retrievalINSERT Data insertionDELETE Data deletionUPDATE Data update

MUSIC START Music starts playingMUSIC STOP Music stops playing

MUSICTYPE CHANGE Change of music typeMUSICPART CHANGE Change of melody partFIND SPECIFIC WORD A specific word appears in lyrics

INPUTVOLUME CHANGE Change of input volumeRECEIVE Input from external equipment

tems. Figure 5 shows the syntax of the ECA rule usedfor Active Karaoke.

In the figure, the Event name describes a name ofan event that triggers this rule. The Definition of vari-ables defines the local variables used within this rule,while the Conditions specifies the condition for execut-ing the following actions. This part is described by thesequence of <Expression 1> <operator> <Expression2>. Expressions 1, 2 describe database attributes orconstant values, and the operator can be ‘=’, ‘>’, ‘<’,‘>=’, ‘<=’, or ‘! =’. The Action describes operationsto be carried out if all conditions are satisfied.

Table 1 shows the events provided by ActiveKaraoke. The SELECT, INSERT, DELETE, UPDATEare events activated when the system retrieves, in-serts, deletes, updates data on the database of ActiveKaraoke. These events are used for ECA rules such as‘When new materials are stored in the database, thesematerials are displayed at a specific point in the mu-sic’. The MUSIC START, MUSIC STOP events areactivated when music starts/stops playing. The MU-SICTYPE CHANGE, MUSICPART CHANGE eventsare activated on the change of the genre (rock, pops..)and the part (bridge, theme) of the song that is playing.The FIND SPECIFIC WORD event is activated whenspecific words appear in the lyrics. The INPUTVOL-UME CHANGE is an event activated by a significantchange of the input volume from microphones. Thisevent is used for ECA rules such as ‘When the singer isexcited, the system displays specific images’.

Table 2 shows the actions provided by ActiveKaraoke. The QUERY action issues a databasequery for management of the database in re-sponse to the change of a situation. The IN-SERT ECA, DELETE ECA actions insert/delete ECA

Table 2: Actions provided by Active KaraokeName Content

QUERY([expression]) Database operationINSERT ECA([description]) Storing an ECA rule

DELETE ECA([identification]) Deletion of ECA rulesENABLE ECA([identification]) Activation of ECA rulesDISABLE ECA([identification]) Deactivation of ECA rules

APPLY EFFECT([identification]) Applying visual effectsDISPLAY GRAPHIC([identification]) Displaying specific images

Table 3: Visual EffectsName Type Content

POS NORMAL POS Normal position (full screen)CHILD POS Dual window (full and small windows)

RANDOM POS Displaying 1/4 size random imagesat four corners by rotation

SMALL POS Displaying a 1/4 size imageat a random position

EFF NORMAL VIS No visual effectEFF GRAY VIS Gray scalingEFF SEPIA VIS Change the color tone of images

to sepia toneEFF FLASH VIS Flash effect

DISP NORMAL DISP No displaying effectSLIDEIN LEFT DISP Slide-in effect from left side.

Same applies to right, top and bottom.WIPEIN LEFT DISP Wipe-in effect from left side.

Same applies to right, top and bottom.CENTER ZOOM DISP Zoom-in effect from center.

POS: Position VIS: Visual effect DISP: Displaying effect

rules into/from the system. Using these actions, thesystem can change its functions dynamically. TheENABLE ECA, DISABLE ECA actions enable/disableECA rules for customizing system behavior withoutdeleting any of the rules. The APPLY EFFECT actionapplies various visual effects to to the displaying images.Table 3 shows the visual effects that can be used in oursystem, which can be categorized into three types: po-sition, visual effect, and displaying effect. This actionspecifies which effects to apply combining effects fromeach category. Moreover, this action assigns the appli-cable scope of the effect shown in Table 4. The DIS-PLAY GRAPHIC action displays specific images. Thisaction also specifies the period of time the image is tobe displayed by same method as the APPLY EFFECTaction.

Additionally, Active Karaoke provides two systemparameters, NEW data and OLD data for each event.The system sets the information shown in Table 5 tothese parameters when the event is activated. Theseparameters can be used in the condition part or theaction part of ECA rules.

Figure 6 shows some examples of ECA rules. In thisfigure, RULE1 converts scenes to sepia colors for cre-ating mood. This rule is activated if the genre of theplaying music is “Ballad”. RULE2 and RULE3 are rulesthat fulfill the demand for special scenes at the climaxof a song. RULE2 initiates the use of a specific imageas the background when the music part becomes “Cli-max,” and RULE3 stops the effect of RULE2 at the endof the “Climax.” RULE4 is an example of processing inresponse to external equipment. If the system receives

Table 4: Specification of Applicable ScopeSpecification Content

ALL Whole span of a songRANGE([start], [end]) Fixed period of time

UPTO([flag specification]) By the time a specific flag becomes onONLY ONE Only next single image

Table 5: Contents of NEW data and OLD dataEvent NEW OLD

SELECT Retrieved data –INSERT Inserted data –DELETE – Deleted dataUPDATE Data after update Data before update

MUSIC START Song information –MUSIC STOP – Song information

MUSICTYPE CHANGE New type Old typeMUSICPART CHANGE New part Old partFIND SPECIFIC WORD position of the word –

VOLUME CHANGE New volume Old volumeRECEIVE Received data –

data from an external equipment named “Maraca,” thisrule displays a specific image for one second. RULE5and RULE6 are rules for displaying random quarter-sized scenes during musical introductions or interludesbecause these musical parts have no lyrics as keywordsfor diciding which scenes to display. RULE7 is an ex-ample of complex effects. When music part becomes a“Bridge,” this rule displays a specific image for five sec-onds with two effects: the flash effect and the zoom-infrom center effect.

3.2 System Structure

Figure 7 illustrates the system structure of ActiveKaraoke. The Active Database Part realizes functionsof active database operations such as ECA rules pro-cessing and scene retrieval. The Event detection mech-anism detects a situation change inside/outside the sys-tem. The ECA rule retrieving mechanism retrievesECA rules for an activated event from an ECA rulebase. If matched rules exist, The ECA rule processingmechanism executes them.

The Interface Part provides graphical user interfacesto select, play and stop songs.

4 Implementation

We implemented a prototype Active Karaoke systemusing Microsoft Visual C++ 6.0 on a DynaBookSS3380(Mobile PentiumII 400Mhz, 128MByte memory). Inaddition, about 20 set of music data are stored in theMusic database, and we use MP3 and MIDI as the for-mat of storing music data. We also stored approxi-mately 1000 scenes in BGV Material database, whereeach scene is a photo image in the JPEG format. Fig-ure 8 shows snapshots of the system with ECA rules

� �CREATE RULE RULE1 ON MUSICTYPE CHANGEWHERE NEW.TYPE = ’Ballad’THEN DOAPPLY EFFECT( ALL, EFF SEPIA, SYS NOWSONG );

� �� �CREATE RULE RULE2 ON MUSICPART CHANGEWHERE NEW.TYPE = ’Climax’THEN DODISPLAY GRAPHIC( UPTO(FLAG(0)),

’C:/graphics/climax.jpg’);

CREATE RULE RULE3 ON MUSICPART CHANGEWHEREOLD.TYPE = ”Climax”

THEN DOFLAG ON(0);

� �� �CREATE RULE RULE4 ON RECEIVEWHERENEW.TYPE = ”maraca”

THEN DODISPLAY GRAPHIC( RANGE(SYS NOWTIMING,1),

’C:/graphics/maracas.jpg’ );

� �� �CREATE RULE RULE5 ON MUSICPART CHANGEWHERENEW.TYPE = ”Interlude”

THEN DOAPPLY EFFECT( UPTO(FLAG(0)),

SMALL, SYS NOWSONG );

CREATE RULE RULE6 ON MUSICPART CHANGEWHEREOLD.TYPE = ”Interlude”

THEN DOFLAG ON( 0 );

� �� �CREATE RULE RULE7 ON MUSICPART CHANGEWHERENEW.TYPE = ’Bridge’

THEN DODISPLAY GRAPHIC( RANGE(SYS NOWTIMING,5),

’C:/climax.jpg’ );APPLY EFFECT( RANGE(SYS NOWTIMING,5),

EFF FLASH, SYS NOWSONG );APPLY EFFECT( RANGE(SYS NOWTIMING,5),

CENTER ZOOM, SYS NOWSONG );

� �Figure 6: Examples of ECA rules

shown in Figure 6. Snapshot 1 is a snapshot displayedat the opening of a song. RULE5 orders the displayof quarter-sized random images transformed into sepiacolor by RULE1. In Snapshot 2, since there is no ruleactivated in this situation, the system displays an im-age which is suited to the lyrics (in this snapshot, theimage is extracted by a keyword ‘Summer’). Snapshot3 is a snapshot during the bridge of a song. In thissituation, RULE7 displays a specific image (climax.jpg)with some effects (FLASH, CENTER ZOOM). In Snap-shot 4, RULE4 displays a specific image (maracas.jpg)because the system receives a data packet from an ex-ternal maraca device.

With regards to the throughput of our system, thescene retrieval process by keywords becomes the bot-

BGV Material

database

ECArule processing mechanism

ECArule retrievingmechanism

Event detection mechanism

Interface Part

ECA Rule

base

Music

database

Related term

database

Active Database Part

External

events

Figure 7: The structure of Active Karaoke

tleneck. This retrieval process consists of two steps:In the first step, the system produces related keywordsfrom a keyword using the related-term dictionary. Itthen retrieves scenes that are appropriate to these key-words. In this research, to resolve this bottleneck, thesystem preassigns keywords with related keywords be-forehand, and stores information in the keyword to key-words database. The bottleneck is broken off to somedegree by using this method. However, the systemhas to remap each time it updates the related-termdatabase. We evaluated the performance of our sys-tem with about 1000 scenes. As a result, the retrievaltime of each scene was less than 0.1 seconds.

The validity of retrieved results depends on keywordsattached to each scene and the related term dictionary.In our prototype system, we use simple keyword match-ing as the retrieving method, therefore, the system can-not always retrieve the best image when considering itsmeanings. We can improve the method of retrieval us-ing other methods. However, it may be interesting whenthe system displays unexpected types of images. Con-sequently, we have to enhance the retrieving methodwithout losing the element of surprise.

5 Related Works

To take example studies of enhanced active databasesystems, there are research works on AMDS (Ac-tive Mobile Database System)[2], SADB (Super ActiveDatabase System)[3]. AMDS and SADB are the en-hanced active database systems for mobile computingenvironments and broadcasting environments. Thesesystems have to process emergent events such as con-nection/disconnection of the mobile host in AMDS, re-ceiving broadcast data in SADB. The concept of activedatabases is effective in such environments. Accord-ingly, Active Karaoke has to manage various emergent

events such as changes in musical parts or voice inputsby a singer. Therefore, it is also effective to develop oursystem based on active database technologies.

6 Conclusion

In this paper, we proposed Active Karaoke, which isa system for generating background scenes for karaokeusing active database technologies. Active Karaokehas various functions for enhancing the enjoyment ofkaraoke.

In the future, we will propose an advanced mecha-nism for scene retrieval using new ideas such as retrievalthat considers previous scenes. Moreover, we will pro-vide more events and actions in the ECA rule formatfor more visual effects and advanced retrieval methods.

Acknowledgements

This research was supported in part by the SpecialCoordination Funds for Promoting Science and Tech-nology from the Ministry of Education, Culture, Sports,Science and Technology, Japan and Grant-in-Aids forScientific Research numbered 13780331 from Japan So-ciety for the Promotion of Science.

References

[1] DKKaraoke Homepage:http://www.dkkaraoke.co.jp/.

[2] T. Murase, M. Tsukamoto, and S. Nishio: “A Sys-tem Platform for Mobile Computing based on an Ac-tive Database,” in Proc. International Symposiumon Cooperative Database Systems for Advanced Ap-plications, vol. 2, pp. 424–427 (1996).

[3] S. Sanguantrakul, T. Terada, M. Tsukamoto, S.Nishio, K. Miura, S. Matsuura, T. Imanaka: “AUser Customized Selection and Categorization forBroadcast Data,” in Proc. of 1999 IEEE Inter-national Workshops on Multimedia Network Sys-tems(MMNS), pp. 596–601 (1999).

[4] All-Japan Karaoke Industrialist Association Home-page: http://www.japan-karaoke.com/．

[5] J. Widom, R.J. Cochrane, and B.G. Lindsay: “Im-plementing Set-oriented Production Rules as an Ex-tension to Starburst,” in Proc. of the SeventeenthInternational Conference on Very Large Data Bases,pp. 275–285 (1991).

Snapshot 1: Opening

Snapshot 2: Routine Play

Snapshot 3: Bridge

Snapshot 4: Maraca Input

Figure 8: An example of Rule Executions