mood lighting system reflectingmusic mood.pdf

Mood Lighting System ReflectingMusic Mood

Chang Bae Moon,1 HyunSoo Kim,1 Dong Won Lee,2

Byeong Man Kim1*1Department of Computer Software Engineering, Kumoh National Institute of Technology, Gumi, Gyeongbuk, South Korea

2Development Team, Hanwha Corporation, South Korea

Received 2 September 2013; revised 12 November 2013; accepted 13 November 2013

Abstract: The emotional impact of music or color can bemaximized if they are used together. This article presentsa mood-lighting system that automatically detects themood of a piece of music and expresses the mood viasynchronized lighting. To do this, the relationshipbetween mood words and colors was analyzed via a webquestionnaire (n 5 202) on moods associated with musicand colors. Data analysis generated lighting scenariosreflecting changes in the mood of music. Each piece ofmusic was divided into several segments using structuralanalysis, with the mood of each segment detected by themood classification module using the neural network. Amatching color was then assigned. The best performanceof our mood classification module was <70%, whichis not sufficient for commercial use; however, thisfigure is high enough to show the potential of thisapproach. VC 2013 Wiley Periodicals, Inc. Col Res Appl, 40, 201–

212, 2015; Published Online 11 December 2013 in Wiley Online

Library (wileyonlinelibrary.com). DOI 10.1002/col.21864

Key words: music mood classification; mood color; moodlighting system; neural network; mood color map; light-ing scenario

INTRODUCTION

In settings such as theaters, caf�es, concerts, and nightclubs,

music can maximize a person’s mood. For example, play-

ing sad music during a tragic scene in a film, or fearful

music during a frightening scene maximizes the intended

mood. Like music, lighting can be a useful tool for express-

ing mood. When the lighting in a caf�e is integrated with the

overall design, this can produce a much better mood in

patrons. Lighting that works together with music is likely

to produce a much stronger mood. For instance, in a con-

cert, emotional feelings are heightened if lighting corre-

sponds to the music that the audience is enjoying.

As combinations of music and lighting promote a

stronger emotional impact, we have developed a system

to produce illumination synchronized with music. To con-

struct such a system, the correlation between music and

colors had to be understood. Therefore, in the article, we

aimed to correlate colors with music indirectly, based on

correlations between mood and color (rather than a direct

correlation between music and color); this system first

identifies the mood of a piece of music, and then chooses

a color that matches the mood.

A number of researchers1–5 have sought to develop

methods to detect the mood of music. These studies have

been based on well-known mood models, such as those

developed by Russell,1 Hevner,2 or Thayer.3 The method

used in this article is based on Thayer’s mood model,

employing a similar approach to existing Thayer-based

models. Our data are unique due to their large volume

and the use of a Korean sample.

Some researchers, including Manav,6 Barbiere et al.,7

Bresin,8 Odbert et al.,9 Spence,10 Wright and Rainwater,11

Valdez and Mehrabian,12 D’Andrade and Egan,13 and Ou

et al.,14 have investigated the correlation between mood

and color. While the results of these studies have

informed our system, we have based our model on our

own correlations between mood and color obtained

through an analysis of a Korean sample, as we suspect

correlations between mood and color may be vary based

on nationality.14 To understand this correlation, the rela-

tionship between mood words and colors was analyzed

via a web questionnaire (n 5 202) on moods associated

*Correspondence to: Byeong Man Kim (e-mail: [email protected])

Contract grant sponsor: Research Fund, Kumoh National Institute of

Technology.

VC 2013 Wiley Periodicals, Inc.

Volume 40, Number 2, April 2015 201

with music and colors. Data analysis generated a lighting

scenario reflecting changes in the mood of music. As we

assumed each musical piece to consist of several parts,

and therefore various moods, the music was divided into

several segments using structural analysis,15 with the

mood of each segment detected by the mood classifica-

tion module using the neural network. For the neural net-

work to learn, 391 acoustic features (e.g., tempo,

chromagram, MFCC, and so forth) were extracted from

each musical segment. We found that the mood classifica-

tion performance of our system declined when all 391

features were used, so we used the regression coefficient

to reduce the number of features used.16 This allowed us

to develop a mood lighting device that displays color

matching the mood of the music being played.

The remainder of this article is organized as follows.

“Related Studies” gives an overview of the pertinent litera-

ture, and “Mapping Music to Mood” describes the methods

used to automatically map the moods associated with a

music segment and analyzes the collected data on moods.

“Mapping Mood to Color” describes the method of map-

ping mood words to colors through an analysis of the col-

lected data on colors associated with mood words, and

“Implementation of Mood Lighting System” describes an

implementation of the system proposed in this article.

Finally, “Conclusion and Future Studies” presents the

conclusion, limitations of the study, and ideas for future

studies.

RELATED STUDIES

Existing emotion models include those by Russell1 (Fig.

1a), Hevner2 (Fig. 1b), and Thayer.3 The Russell and

Fig. 1. Mood models. (a) Russell Model. (b) Hevner Model. (c) Thayer’s Two-Dimensional Model.

202 COLOR research and application

Hevner models use adjectives to describe emotions, which

can result in ambiguity when adjectives have duplicate

meanings. For this reason, in the article, we used

Thayer’s two-dimensional model, in which mood or emo-

tion are expressed by a combination of arousal and

valence (AV) values. Arousal refers to the strength of

stimulation that listeners feel (i.e., weak or powerful) and

valence refers to the intrinsic attractiveness (positive

valence) or aversiveness (negative valence). Figure 1c

depicts Thayer’s two-dimensional mood model, as well as

the relationships among 12 adjectives used to describe

moods and emotions.

Liu et al.17 has presented a music mood recognition

system that used a fuzzy classifier to categorize a Strauss

waltz into five classes. Features such as tempo, strength,

pitch change, note density, and timbre are used. Katayose

et al.4 proposes a sentiment–extraction system for pop

music; in this system, monody sound data are first con-

verted into music code, from which melody, rhythm, har-

mony, and form are extracted. These two systems are

useful in themselves, but they use musical instrument dig-

ital interface or symbolic expression, as it is difficult to

extract useful features from sound data. However, much

of the sound present in the real world cannot be

expressed through symbols; no system exists that can cor-

rectly translate such sound data into symbolic expres-

sions.18 This limitation necessitates a system that can

directly detect mood from sound data.

Feng et al.19 proposed a method of classifying moods

into four groups—happiness, sadness, anger, and fear—

using tempo and articulation features. Li and Ogihara20

proposed a method of detecting mood using timbre, tex-

ture, rhythm, and pitch features, using 13 adjective groups

based on checklists by Hevner21 and Farnsworth22 as

mood classes. Yang et al.23 used a fuzzy-based method to

solve the ambiguity of expression that can occur when

only a single mood is allowed; they expressed musical

mood as a mix of several moods, denoted as separate

numerical values. However, Yang et al.24,25 noted that

this method might fail to take into account subjective

individual preferences, which would be necessary for per-

sonalized service. To solve this problem, rather than using

a single mood class, the authors used AV values com-

posed of two real number values between 21 and 1 on

each axis of Thayer’s two-dimensional mood model.

They used two regressors to model AV values collected

from subjects, and suggested a personalized detection

method that characterizes users into “professional” or

“non-professional” groups depending on their degree of

understanding of music.

A number of studies5,7–9,26 have investigated the asso-

ciation between music and color. Barbiere et al.7 asked

subjects to assign 11 basic colors according to music with

two types of mood, “happy” or “sad.” Brighter colors

were usually associated with happy songs, while more

muted colors were usually assigned to sad songs. Bresin8

asked subjects to rate how well each of 8 colors in each

of their 3 shades corresponded with 12 music pieces

expressing different emotions. Different hues were associ-

ated with different emotions; furthermore, dark colors

were associated with minor tonality and light colors with

major tonality. Odbert et al.9 surveyed the mood of music

using Hevner’s model and asked subjects what color they

associated with the music. Through analyzing the rela-

tionship between the mood of music and the colors sug-

gested by that music, They demonstrated that subjects

who disagree on the mood of a piece of music tend to

also disagree on the colors associated with that music.

These results were very similar to those obtained by stud-

ies in which subjects were asked to name the color best

fitting certain mood words. Palmer et al.26 provided

experimental evidence that music and colors were medi-

ated by emotional association in two cultures, US and

Mexico, by showing that there were strong correlations

between the emotional associations of the music and

those of the colors chosen to go with the music. They

showed that faster music in the major mode made partici-

pants choose more saturated lighter colors and yellower

colors whereas slower. They also showed that minor

music produced the opposite pattern, that is, desaturated,

darker, and bluer colors were chosen for minor music.

Some researchers6,11–14 have investigated the relation-

ship between color and mood. Manav6 defined this rela-

tionship using adjective mood words. In this study,

subjects were provided with 41 colors and were asked to

select which of 30 provided adjectives (e.g., vivid, bor-

ing, cold, warm, exciting, fearful, mysterious, peaceful,

and relaxing) best matched them. Manav further ana-

lyzed responses for 10 colors and looked at associations

between subjects’ education level, age, and gender.

Based on these findings, he recommended certain colors

be used for bedrooms, bathrooms, and children’s room.

Valdez and Mehrabian12 investigated the relationship

between mood and color using the Pleasure–Arousal–

Dominance (PAD) emotion model, providing PAD-value

prediction equations with the parameters of hue, satura-

tion, and brightness. Valdez and Mehrabian12 demon-

strated strong and reliable relationships between

emotional reactions and each of brightness and satura-

tion, but only a weak relationship with hue. Ou et al.14

demonstrated that color preference can be determined

using three color-emotion scales (clean–dirty, tense–

relaxed, and heavy–light), with clean–dirty being the

predominant scale. They also showed that color prefer-

ence can be determined by the three color-appearance

attributes of hue, lightness, and chroma; the most dis-

liked color was found to be at the hue angle of 105�

with the chroma value of 31.

Previously,27 we investigated the relationship between

mood and color. However, the findings of our prior study

were not useful for informing our lighting system, as the

relationship focused on users’ individual musical prefer-

ences. As our mood-lighting system is intended to be

installed in public spaces such as parks, plazas, and

squares, we needed a model that mapped mood to color

without considering individual users’ preferences.


MAPPING MUSIC TO MOOD

In this article, to find a color that matched the music, first

the mood of the music was determined, and then the

color corresponding to that mood was chosen. This sec-

tion describes the method of mapping music to mood; the

section after this describes the method of mapping mood

to color. As shown in Fig. 2, the mood mapping process

of music consisted of two phases: one was mood training

phase and the other mood identification phase. In the

mood training phase, features of each music segment of

Moon et al.’s27 were extracted by MIRtoolbox28 and then

were reduced by the R2 method (see "Mapping Music

Segments to Moods" section); using the reduced feature

sets of the music segments and their mood information,

the relationship between mood and music (or mood map-

ping model or shortly mood model) was modeled via a

neural network. The music segments of Moon et al’s27

had been obtained by dividing the music into several seg-

ments via Levy et al.15 method and their mood informa-

tion had been also collected from several subjects in our

previous work.27 In the mood identification phase, a new

music piece was divided into several segments via Levy

et al.15 method; their features were extracted and reduced;

the mood of each segment was determined by inputting

its reduced features into the mood model built in the

training phase.

Collecting the Mood of Music Segments from Subjects

To understand the relationship between mood and

music, we used Moon et al’s27 mood data set. This data

set had been collected for 3 days in a room with only one

dark glass window on one side, between the hours of

approximately 10:00 am to 5:00 pm. The subjects were

189 general participants. A total of 281 musical segments

(each of approximately 12 s) were extracted from 101

music pieces and were used for mood collection. From

this set of 281 segments, 47 were selected randomly and

played for the 189 general participants. All participants

used headsets to prevent interference from non-study-

related noises.

Figure 3 shows the mood distribution of some sample

music pieces: note that approximately three music seg-

ments were extracted from each music piece, and that the

moods of these segments may differ. Distributions of 101

pieces of music can be summarized and classified into six

types. The first type is a piece of music which has one

predominant mood (Fig. 3a); the second type is for a

piece in which all moods are similar (Fig. 3b); the third

type contains a wide distribution of moods covering two

quadrants (Fig. 3c); the fourth type contains different

moods, all of which are in one quadrant (Fig. 3d); the

fifth type contains different moods with a wide range of

distribution (Fig. 3e); and the sixth type has high frequen-

cies of several moods (Fig. 3f).

In Fig. 3, different colors denote different segments of

the same piece of music; there were an average of three

segments per piece. This analysis supports our assumption

that individual music pieces contain several moods, mak-

ing it necessary to change the lighting color when playing

music.

Mapping Music Segments to Moods

To map a music segment to a mood automatically, it is

necessary to learn the relationship between its musical

features and its mood. In this article, features of music

segments were extracted using MIRtoolbox.28 These fea-

tures can be divided into five categories: dynamics,

rhythm, timbre, pitch, and tonality. Dynamics includes

root mean square (RMS) energy, and rhythm includes

fluctuation summary, tempo, and attack times. Timbreincludes zero cross rate, brightness, and roll-off, as well

as spectral centroid, spread, skewness, kurtosis, entropy,

and flatness. Pitch refers to pitch and chromagram, and

tonality includes the clarity, mode, and harmonic change

detection function. We used the 391-dimensional vector

produced by the mirfeatures function of MIRtoolbox,

which calculates statistics such as the mean, standard

deviation, and slope, as well as the frequency, amplitude,

and entropy of periods, instead of the complete features

themselves.

If all 391 features of the feature vector are used, map-

ping performance may decrease due to the effect of noise.

For this reason, we chose some noise features using the

well-known dimensional reduction method, R2 reduc-

tion.29 In our experiments, we used the 50 features with

Fig. 2. Music mood mapping process.


the largest r2 values [see Eq. (1)] (Table I). The perform-

ances of these selected features are given in the section

“Mood Mapping Performance.” The term r2 was calcu-

lated as follows:

r25

nPni51

xiyi

� �2

Pni51

xi

� � Pni51

yi

� �ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffinPni51

x2i

� �2

Pni51

xi

� �2s ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

nPni51

y2i

� �2

Pni51

y2i

Pni51

yi

� �2s

0BBBB@

1CCCCA

2

(1)

where r2 is the regression coefficient, xi is the ith feature

vector, and yi is the class number of the ith feature vector.

In this article, the model used to map the musical fea-

tures to mood was automatically generated using the neu-

ral network approach, a well-known machine-learning

technique, and based on a training dataset consisting of

the musical features of music segments and their associ-

ated moods. Subjects might assign different moods to the

same music segment, but it is necessary to define one rep-

resentative mood per segment; in the training dataset,

only the representative mood of each music segment is

given. In this article, the representative mood of each seg-

ment was calculated using the definition in Moon et al.’sstudy.27

Figure 4 shows the structure of the neural network

used in this study. The neural network consisted of 50

nodes in the input layer and 4 nodes in the output layer.

The feature vector consisting of 50 musical features, as

described above, was used as the input for the neural net-

work, and a four-dimensional vector was the output. The

output vector (1, 0, 0, 0) indicates the first quadrant of

the AV model, (0, 1, 0, 0) indicates the second quadrant,

(0, 0, 1, 0) indicates the third quadrant, and (0, 0, 0, 1)

indicates the fourth quadrant. Note that the number of

output nodes should be 12 rather than 4, as there were 12

possible moods for participants to choose from. However,

our sample of 281 musical segments meant that there was

only an average of 23 segments per mood, with fewer

Fig. 3. Mood distribution of music. (a) Case 1: Music 37. (b) Case 2: Music 13. (c) Case 3: Music 95. (d) Case 4: Music74 (e) Case 5: Music 1. (f) Case 6: Music 28.


than 10 segments for some moods, an insufficient number

to learn neural networks. For this reason, the 12 moods

were categorized into four groups corresponding with the

quadrants of Fig. 1c.

Mood Mapping Performance

To evaluate the performance of the mapping model,

leave-one-out cross-validation (LOOCV)30 was used. In

LOOCV, a single example from a set of examples is used as

the validation data, and the remaining examples are used as

the training data. This is repeated such that each example in

the set is used once as the validation data. In this article, we

chose 164 music segments covering all moods evenly as

examples among the total 281 samples. Precision which is

the number of correctly classified positive examples divided

by the number of examples labeled by the system as positive

was used as the measure of classification performance.

The performance of the mapping (or classification)

model might be dependent on the number of hidden

nodes or the number of iterations of learning. So, we con-

ducted experiments varying the number of hidden nodes

from 2 to 25 and employing two different iterations

(3000 times and 4000 times) in the learning stage of the

neural network. As shown in the previous section, for

performance reasons, we did not use 391 features, but

used only the top 50 features after sorting the features

with the regression coefficient calculated by Eq. (1). The

experimental results are shown in Fig. 5. With 3000

learning iterations, we obtained the best performance

(66.46%) using 17 hidden nodes (Fig. 5b). With 4000

learning iterations, we obtained the best performance

(65.24%) using 7 hidden nodes (Fig. 5a). From the

results, we could conclude that the number of hidden

nodes and the number of iterations of learning did not

contribute to the performance improvement of our model

impressively. The performance we got is not suitable for

commercial use, but we believe it confirms the promise

TABLE I. Top 50 musical features used in the experiments

Category Subcategory Sub-sub category

DYNAMICS RMS energy SDRHYTHM Attack slope Period entropy

Attack time Mean, period entropyTempo SD

TIMBER Spectrum Mean, SDSpectral irregularity Mean, SDSpectral kurtosis Mean, SDZero crossingrate Mean, SDBrightness MeanEntropy of spectrum MeanRolloff (85%) MeanRolloff (95%) MeanSpectral centroid MeanSpectral flatness MeanSpectral flux MeanRoughness Mean, Period AmpSpectral skewness SDMel-frequency cepstral

coefficients (MFCC)Mean (1, 8, 9, 10), Period Freq

(9, 11, 12, 13)SD (6, 7, 8, 9, 10, 11, 12, 13)Delta MFCC (DMFCC) Period Amp (2), Std (4)Delta Delta MFCC (DDMFCC) Std (4)

TONAL Centroid of chromagram MeanChromagram MeanHarmonic change detection Function Mean, Period AmpKeyclarity Mean, StdMode Period AmpPeak chromagram Peak Pos Mean

Fig. 4. Structure of neural network to learn and determinethe mood of music segments. Fig. 5. Performance of mapping music to mood.


of our approach: our future research will focus on

improving the mood mapping performance.

MAPPING MOOD TO COLOR

To map mood to color, the relationship between mood

and color was analyzed using Moon et al’s27 color data

set. This data set was collected by asking subjects to

select a color matching a mood word, under the same

conditions described in the section on collecting moods

of music. Based on subsequent analysis, a representative

color was chosen for each mood word, which was used to

generate mood lighting scenarios.

Representative Color of Mood

Since different subjects might associate different colors

with given mood words, we needed to define the repre-

sentative color of a mood word. For this, a mood color

map representing frequencies of colors that subjects

selected as the color matching the given mood was con-

structed for each mood word (e.g., Fig. 6 presents the

color map of “Nervous”). To create a lighting scenario,

the representative color (i.e., that with the highest selec-

tion frequency) was first selected from the mood color

map. In Fig. 6, for example, the representative color is

dark red, selected by 11 subjects.

The representative colors for the 12 mood words are

shown in Table II. The mood words belonging to Group

1 (Excited, Happy, and Pleased) are mapped with bright

red, bright red, and bright yellow, respectively; Group 2

(Angry, Annoying, and Nervous) with red, red, and dark

red, respectively; Group 3 (Sad, Bored, and Sleepy) with

dark yellow, bright blue, and bright yellow, respectively;

and Group 4 (Calm, Peaceful, and Relaxed) with bright

green, bright light green, and bright green, respectively.

Comparisons with results from related studies are given

in Table III.

The colors defined in Table II should be used when

mapping mood words to colors; however, the music was

mapped to 4 mood groups in our experiments, as opposed

to 12 moods. To accommodate this methodological

change, the color map for each mood group was recon-

structed by adding the color maps of moods belonging to

each group; from this, the color with the maximum fre-

quency was selected as the representative color of that

mood group. Group 1 was yellow (RGB: 255-255-51,

HSV: 42–204-255), Group 2 was red (RGB: 204-0-0,

HSV: 0–255-204), Group 3 was bright blue (RGB: 204-

204-255, HSV: 170-51-255), and Group 4 was bright

green (RGB: 153–255-153, HSV: 85–102-255).

Analyzing the Color Distribution of Mood Words

In Moon et al.’s27 color data set, the average number

of colors per mood was 154, for a total of 1848 colors.

For our analysis, we summarized the color of each mood

word as a percentage (Fig. 7). As was described in the

section “Mapping Music Segments to Moods,” 12 mood

words were grouped into 4 groups: Group 1 (Excited,

Happy and Pleased); Group 2 (Annoying, Angry and

Nervous); Group 3 (Sad, Bored and Sleepy); and Group 4

(Calm, Peaceful and Relaxed). Figure 7 summarizes the

color distribution for each group. After creating this color

distribution, we examined the relationship between colors

and mood words. For simplicity, we grouped colors into

six groups by their hue values, such that colors with hues

of 230–29 were mapped onto Red, 30–89 onto Yellow,

90–149 onto Green, 150–209 onto Cyan, 210–269 onto

Blue, and 270–329 onto Magenta.

Fig. 6. Mood color map of “Nervous.”

TABLE II. Representative colors of 12 moods

No Mood Color R G B H S V

1 Excited Bright red 255 51 51 0 204 2552 Happy Bright red 255 153 153 0 102 2553 Pleased Bright yellow 255 255 51 42 204 2554 Angry Red 204 0 0 0 255 2045 Annoying Red 255 0 0 0 255 2556 Nervous Dark red 153 0 0 0 255 2537 Sad Dark yellow 102 102 0 42 255 1028 Bored Bright blue 204 204 255 170 51 2559 Sleepy Bright yellow 255 255 204 42 51 25510 Calm Bright green 153 255 153 85 102 25511 Peaceful Bright

light green204 255 153 63 102 255

12 Relaxed Bright green 153 255 153 85 102 255


As shown in Fig. 7, red was most often selected for

Group 1 mood words; however, yellow was also highly

selected for happy and pleased, and blue was frequently

associated with excited. Red was also most often selected

for Group 2 mood words; the mood word with the highest

percentage of Red among the three words in this group

was angry. Blue and yellow were the main colors for

Group 3; Sleepy was frequently associated with yellow

while sad and bored had relatively equal associations

between blue and yellow. The main colors of Group 4

were yellow and green.

We also examined the relationship between the bright-

ness and saturation of colors and the associated mood

words. Figure 8 represents the brightness and saturation

of mood words as a percentage: the x-axis indicates the

numerical value of brightness and saturation obtained

from the default color table given by Java31 (top row-

5 saturation; bottom row 5 brightness). Group 1 mood

words were associated with brighter and more highly sat-

urated colors (Fig. 8a); Group 2 mood words with less

bright but highly saturated colors (Fig. 8b); Group 4 with

brighter but less saturated colors (Fig. 8d). For Group 3,

no pattern of association between brightness or saturation

and mood words was detected (Fig. 8c).

IMPLEMENTATION OF MOOD LIGHTING SYSTEM

The configuration of the emotional lighting system repre-

senting the mood of music is shown in Fig. 9. A PC

TABLE III. Comparisons of color mapping

Color Proposed Manav6 Barbiere et al.7 Bresin et al.8 Odbert et al.9

Red Angry, annoying Happy Anger, jealousy ExcitingBright red Excited, happy HappyDark red NervousBlue Hygiene, pure Happy Love, fear TenderBright blue

(light blue)Bored Calm, peaceful,

modern, relaxingHappy

Yellow Hygiene, pure Happy Happiness, pride PlayfulBright yellow

(light yellow)Pleased, sleepy Simple, classic, plain Happy

Dark yellow SadGreen Hygiene, pure Happy LeisurelyBright green Calm, relaxed HappyBright light green PeacefulBlack SadGray SadNear white Hygiene, pure, plain,

simpleCyan CuriosityOrange Contentment, shame GayPink Warm, romantic,

enjoying, cheerful,striking

Purple SolemnViolet Sadness, love,

tenderness, disgust

Fig. 7. Color distribution according to AV model area. (a) Color Distribution of Group 1. (b) Color Distribution of Group 2.(c) Color Distribution of Group 3. (d) Color Distribution of Group 4.


sends the lighting scenario through a network to a control

board, which in turn controls the lighting in accordance

with the scenario received. At the same time, the PC

plays the music (Fig. 9b). The system consists of a mood

color DB (Database) and five modules: music segmenta-

tion, music playing, mood extraction, the creation of

mood lighting scenarios, and the mood lighting module.

The mood lighting module is installed on the control

board, and the other components are installed on the PC.

Figure 9a shows the structure of the system that col-

lects the moods of music and the colors of moods. The

server selects music segments by analyzing the music’s

structure and sends these structures or mood words to cli-

ents. On the client end, a user is required to input the

moods of music segments or colors of moods. The input

data are saved in the music mood or mood color database

and used later, when needed.

Music Segmentation Module

To acquire music segments, the structure of the music

is first analyzed. Then, using the information about the

music’s structure obtained through structured analysis,24

the music is segmented into several pieces, each with

similar acoustic features. In music structure analysis, sim-

ilar segment clustering is used based on the state

sequence.15 Levy et al.15 extracted a music feature vector

and calculated a timbre-type sequence; finally, the music

structure information is identified through a timbre-type

soft k-means clustering algorithm. The music is then seg-

mented according to its structure.

Fig. 8. Brightness and saturation distribution for groups of mood words. (a) Group 1 brightness and saturation distribu-tion. (b) Group 2 brightness and saturation distribution. (c) Group 3 brightness and saturation distribution. (d) Group 4brightness and saturation distribution.

Fig. 9. Mood lighting system reflecting the mood of music. (a) Mood collection system configuration. (b) Mood lightingsystem configuration. (c) Internal configuration of system.


Mood Extraction Module

To determine the mood of a music segment (Fig. 10),

391 features of a music segment were extracted and

reduced using the R2 reduction method.29 In this module,

a mood model is constructed by the neural network, using

data consisting of the reduced features of music segments

and their representative moods, by which the mood of a

new music segment is determined.

Mood Lighting Scenario Creation Module

Figure 11 shows the process of generating a music

lighting scenario. When a piece of music is entered into

the system, its structure is analyzed and it is separated

into segments by the music segmentation module. Mood

segments are then determined by the mood extraction

module. The lighting scenario for a music segment is

generated using its mood information and the color map

of the mood obtained from the mood color information

database. The overall scenario of the selected piece of

music is obtained by combining the scenarios of each

music segment sequentially, and is saved in the lighting

scenario file.

The lighting scenario of a music segment is generated

by taking into account the mood of the current and

upcoming music segments. A sequence of HSV color vec-

tors is generated so that the color changes smoothly from

the representative color of one music segment to the next.

For example, if the mood changes from Angry to Annoy-

ing (Fig. 12a), HSV color vectors (Fig. 12b) are generated

to display the colors located along the connecting dotted

line, with the time interval calculated by considering the

music playing time and the number of colors to be dis-

played during that interval.

Music Playing Module

In the PC, a lighting scenario file is sent to the control

board before the music is played. The user interface for

playing music is shown in Fig. 13a, and the music play

list is shown in Fig. 13b. Music is played in the order of

the play list and, once determined, the order cannot be

modified. In this article, numbers are used for identifying

music instead of file names due to copyright restrictions.

The user interface (Fig. 13c) enables administrators to

check the status of the lighting device. If there is no prob-

lem with a lighting device, the color of the bulb is indi-

cated in white. If there is a problem, the bulb is black.

The number on the lower part of the bulb (963 in Fig.

13d) is the identifier of the lighting module. The user

interface (Fig. 13d) allows users to check the communica-

tion status of the network, which is shown to the user in

real time. The communication status is also recorded in

the log file for reviewing later.

Mood Lighting Module

The mood lighting module consists of an RGB light-

emitting diode (LED) matrix and a control board (Fig.

14). The RGB LED matrix expresses color by receiving

the RGB signal from the control board. The control board

sends data to and from the host system, in addition to

controlling the RGB LED matrix.

The control board consists of an micro controller unit

(MCU), pulse width modulation (PWM) GENERATOR,

Fig. 10. Process of learning and determining the mood of a music segment.

Fig. 11. Process of creating a mood lighting scenario. (a)Lighting scenario. (b) Scenario saving vector.

Fig. 12. Example of creating a lighting scenario. (a) Music player. (b) Music playlist. (c) Status of lighting devices.(d) Communication status.


static random access memory (SRAM), real-time clock

(RTC), wired LAN and 802.11. Each component is

described below:

� MCU: controls all devices

� PWM GENERATOR: controls the LED of the RGB

LED matrix and expresses color

� SRAM: saves RGB data (lighting scenario)

� RTC: provides a real-time clock

� Wired LAN, 802.11: communicates with the host system

The control board is shown in Fig. 15. Figure 15a

shows the back of the control board; the RGB LED

MATRIX (Fig. 15b) is mounted on the front of the con-

trol board using the supporters. For data communication

with the host system, 10 Base-T or 802.11.b/g is used,

and the color of lighting is controlled by the LED driver.

The cross shape at the center of the RGB LED matrix

allows heat generated from the LED to be expelled using

the cooling fan mounted at the back.

CONCLUSION AND FUTURE STUDIES

Both music and lighting can be used to express mood;

moods can therefore be maximized by combining music

and lighting. This article proposed a system that automati-

cally segments music into several pieces, extracts their

moods, and displays the color matching each mood via an

Fig. 13. Host–user interface.

Fig. 14. Mood lighting module. (a) Rear. (b) Front.

Fig. 15. Hardware of the mood lighting system.


LED lighting device. To do this, we analyzed the associa-

tions between mood words and colors using Moon

et al.’s27 mood and color datasets. We observed the color

distribution associated with each mood and set a represen-

tative color for each mood word.

In this article, we mapped music to color indirectly,

first determining the mood of a music segment and then

selecting a color that matched that mood. To determine

the mood of a music segment, we took a similar approach

with prior approaches based on the Thayer model. How-

ever, the data used in the article differed from their data-

set in that music segments were automatically generated

through a structural analysis, not manually. Furthermore,

our mood data set was much larger and was collected

from Korea. Finally, prior studies have treated musical

pieces as unchanging in mood, while our model separated

music into short segments that could have different

moods.

We obtained our own data on the correlations between

mood and color based on a large sample of Korean par-

ticipants instead of using the correlations previously

determined by researchers such as Manav,6 Barbiere

et al.,7 Bresin,8 Odbert et al.,9 Spence,10 Wright and

Rainwater,11 Valdez and Mehrabian,12 D’Andrade and

Egan,13 and Ou et al,14 whose results differed greatly

from ours. This discrepancy may reflect cultural differen-

ces in samples of different nationalities; however, our

findings should not be generalized to the Korean popula-

tion as a whole, as our sample was mostly university

students.

The prototype proposed in this article lays the founda-

tion for a commercial system; however, before such a

system can be developed, mood classification perform-

ance must be improved. The model also needs to be

expanded to cover 12 distinct moods, as opposed to 4

mood groups, as was done in the current study. We

assumed that the algorithm successfully extracted the

mood from the music, and a color was chosen to match

that mood. So, in the near future, we need to show the

evidence that participants agree that the color matches the

music. Furthermore, we need to get a more continuous

map from color to music through mood by allowing

music to represent different moods at once and colors to

represent the multiple moods as well.

1. Russell JA. A circumplex model of affect. J Personality Social Psychol

1980;39(6):1161–1178.

2. Hevner K. Experimental studies of the elements of expression in music.

Am J Psychol 1936;48(2):246–268.

3. Thayer RE. The Biopsychology of Mood and Arousal. New York:

Oxford University Press; 1989.

4. Katayose H, Imai M, Inokuchi S. Sentiment extraction in music. Inter-

national Conference on Pattern Recognition, 1988. p 1083–1087.

5. Lee JI, Yeo D-G, Kim BM, Lee H-Y. Automatic Music Mood Detec-

tion through Musical Structure Analysis. International Conference on

Computer Science and its Application CSA, jeju, Korea, 2009. p 510–

515.

6. Manav B. Color-emotion associations and color preferences: a case

study for residences. Color Res Appl 2007;32(2):144–150.

7. Barbiere JM, Vidal A, Zellner DA. The color of music: correspondence

through emotion. Emp Stud Arts 2007;25(2):193–208.

8. Bresin R. What is the color of that music performance? Proceedings of

the International Computer Music Conference, 2005. p 367–370.

9. Odbert HS, Karwoski TF, Eckerson AB. Studies in synesthetic think-

ing: I. Musical and verbal associations of color and mood. J Gen Psy-

chol 1942;26:153–173.

10. Spence C. Crossmodal correspondences: A tutorial review. Attent Per-

cept Psychophys 2011;2011(73):971–995.

11. Wright B, Rainwater L. The meanings of color. J Gen Psychol 1962;

67:89–99.

12. Valdez P, Mehrabian A. Effects of color on emotions. J Exp Psychol

Gen 1994:123(4):394–409.

13. D’Andrade R, Egan M. The colors of emotion. Am Ethnolog 1974;

1(1):49–63.

14. Ou L-C, Luo MR, Woodcock A, Wright A. A study of colour emotion

and colour preference. Part III: colour preference modeling. Color Res

Appl 2004;29(5).

15. Levy M, Sandier M, Casey M. Extraction of high-level musical struc-

ture from audio data and its application to thumbnail generation. Pro-

ceedings IEEE International Conference Acoust, Speech, Signal Process

(ICASSP), Vol. 5, Toulouse, France, May 2006. p 13–16.

16. Kim JW, Kim HJ, Kim BM. Determination of usenet news groups by

fuzzy inference and neural network. Proceedings of the Korea Fuzzy

Logic and Intelligent Systems Society Conference, 2004.

17. Liu D, Zhang NY, Zhu HC. Form and mood recognition of Johann

Strauss’s waltz centos. Chin J Electron 2003;12(4):587–593.

18. Scheirer ED. Music-Listening Systems, Ph.D. Thesis, MIT Media Lab,

2000.

19. Feng Y, Zhuang Y, Pan Y. Popular music retrieval by detecting mood.

Proceedings of the 26th International ACM SIGIR Conference on

Research and Development in Information Retrieval, Toronto, Canada,

2003. p 375–376.

20. Li T, Ogihara M. Detecting emotion in music. Proc. ISMIR 2003,

2003.

21. Hevner K. Expression in music: a discussion of experimental studies

and theories. Psychol Rev 1935;42:186–204.

22. Farnsworth PR. The Social Psychology of Music. New York: Dryden

Press; 1958.

23. Yang YH, Liu CC, Chen HH. Music Emotion Classification: A Fuzzy

Approach. Proceedings of ACM Multimedia 2006 (ACM MM’06),

Santa Barbara, CA, USA, 2006. p 81–84.

24. Yang YH, Su YF, Lin YC, Chen HH. Music emotion recognition: The

role of individuality. Proceedings of ACM International Workshop on

Human-centered Multimedia (ACM HCM), 2007. p 13–21.

25. Yang YH, Lin YC, Su YF, Chen HH. A regression approach to music

emotion recognition. IEEE Trans Audio Speech Lang Process 2008;

16(2):448–457.

26. Palmer SE, Schloss KB, Xu Z, Rrado-Leon LR. Music-color associa-

tions are mediated by emotion. Proc Natl Acad Sci 2013;110(22):8836–

8841.

27. Moon CB, Kim HS, Lee HA, Kim BM. Analysis of relationships

between mood and color for different musical preferences. Color Res

Appl 2013.

28. Lartillot O. MIRtoolbox 1.2.4, Finnish Centre of Excellence in Interdis-

ciplinary Music Research, March, 18th, 2010.

29. William Mendenhall M, Beaver RJ, Beaver BM. Introduction to Proba-

bility and Statistics. Independence, KY: Cengage Learning; 2008.

30. Croft B, Metzler D, Strohman T. Search Engine: Information Retrieval

in Practice. Upper Saddle River, NJ: Pearson Education, 2010.

31. Oracle, JAVA API. Available at: http://docs.oracle.com/javase/1.4.2/

docs/api/java/awt/Color.html. Last accessed 27 February 2013.


mood lighting system reflectingmusic mood.pdf

Documents