Modelling human preference in evolutionary art

Aniko Ekart1, Divya Sharma2, and Stayko Chalakov1

1 Aston UniversityAston Triangle, Birmingham B4 7ET, UK

Email: a.ekart@aston.ac.uk, stayko16@yahoo.com2 IIT Ropar, India

Email: divyas@iitrpr.ac.in

Abstract. Creative activities including arts are characteristic to hu-mankind. Our understanding of creativity is limited, yet there is sub-stantial research trying to mimic human creativity in artificial systemsand in particular to produce systems that automatically evolve art appre-ciated by humans. We propose here to model human visual preference bya set of aesthetic measures identified through observation of human se-lection of images and then use these for automatic evolution of aestheticimages.

Keywords: aesthetic measure, human preference modelling, genetic pro-gramming, interactive vs automatic evolution

1 Introduction

Ever since the invention of the first computing device, humanity has been think-ing about using them to perform creative activities. Producing aestheticallypleasing pieces of art is certainly one such creative activity. Beginning withthe pioneering work of Dawkins [7] and Sims [19], over the past twenty yearsa lot of effort was spent on generating increasingly more effective evolutionaryart systems that produce aesthetic artworks. Successful examples attracting sub-stantial public attention include the Electric Sheep [10], the NEvAr system [14]and the Painting Fool [3].

The majority of evolutionary art systems are either interactive (for example[21]) or automatic (for example [1, 9]). Interactive systems tend to generate moreaesthetic artworks, as their driving force is human selection, but at the sametime need a lot of effort on the part of the human, may incur user fatigue andcould be inconsistent over time. Automatic systems have the advantage of abuilt-in automatic fitness evaluation, so the human effort is reduced; however,the aesthetics of the resulting artworks may suffer as the automatic evaluationhas not been perfected yet. To overcome the disadvantages and also combinethe advantages of both approaches, Machado et al. propose partially interactiveevolution [15], where the human user’s contribution is much reduced comparedto the fully interactive approach, but the human still guides the evolution.

Substantial efforts in evolutionary art research have been dedicated to study-ing and devising good aesthetic measures [8, 13, 17, 18]. It is generally agreed that

formulating a universally valid and acceptable aesthetic criterion is not withinour reach. Achieving automatic evolution that produces aesthetic images to theliking of the human user very strongly depends on the understanding of the par-ticular user’s aesthetic values. A recent study by Li and Hu [12] suggests usingmachine learning to learn the differences between aesthetic and non-aestheticimages, as indicated by image complexity and image order. Colton [4] producesnew rules for forming fitness functions through the use of an inference engine.Greenfield proposes the technique of evolutionary refinement [11] to encourageaesthetic pattern formation through stages and concludes that ”evolution instages with radical changes in fitness criteria may be a profitable evolutionaryexploration strategy”.

Our contribution complements these previous approaches by considering fourestablished aesthetic measures in interactive evolutionary art to model humanpreference. We monitored how these measures evolved over the generations whendifferent users interacted with a simple evolutionary art system and fully drovethe selection process. We found that a combination of aesthetic measures mod-els user preference suitably well. We consequently employed this combination(MC and BZ) to automatically evolve further images starting from the result ofinteractive evolution.

2 Aesthetic measures

We study the evolution of four well-known aesthetic measures in an attempt tomodel human selection in interactive evolutionary art. Measure R is based onRalph’s work, measure MC is based on Machado and Cardoso’s work, measureBZ on Birkhoff and Zurek’s work and finally measure S on Shannon entropy.

2.1 Aesthetic measure R

This aesthetic measure is based on the mathematical model proposed by Ralph[18].After analyzing hundreds of examples of fine art, it was found that many worksconsistently exhibit functions over colour gradients that conform to a normal orbell curve distribution.

The colour gradient for each pixel is computed as:

|∇ri,j |2 =(ri,j − ri+1,j+1)2 + (ri+1,j − ri,j+1)2

d2

where ri,j is the value of the Red component for pixel (i, j) and d is a scalingfactor which is taken to be 0.1% of the diagonal length, as suggested by Ralph’smodel [18] (leading to the value d2 = 3.277∗10−2). ∇gi,j and ∇bi,j are computedsimilarly for the Green and the Blue colour components.

The overall gradient, or the stimulus, of each pixel is calculated as follows:

Si,j =√|∇ri,j |2 + |∇gi,j |2 + |∇bi,j |2.

Next, the viewer’s response to each pixel is computed as

Ri,j = log(Si,j/S0).

The range of values for Ri,j is [0, 1). R can never become negative. The minimumvalue of 0 corresponds to the case when there is no change in colour at a pixelat all; if there is no stimulus the response is 0. Si,j can never be less than S0 dueto the scaling factor. S0 is the detection threshold taken to be 2, as suggestedby Ralph’s model of aesthetics. If Si,j = 0 (no change in colour at a pixel), it isignored.

The mean µ and standard deviation σ of the response values are calculatedusing the response values themselves as weights because the probability thata viewer pays attention to a detail of an image is considered proportional tothe magnitude of the stimulus that resides at that detail. A histogram is builtnext to judge how close the distribution of response values is to the bell curvedistribution. The ”bins” will each represent an interval of size σ/100. Then theprobability that a response value falls in a given bin is computed. This is repeatedfor all the bins by going through all the Ri,j values where each Ri,j updates itscorresponding bin using a weight of Ri,j .

Then, the deviation (D) from the normal distribution is computed as follows:

D =∑

i

pi log(pi

qi

)where pi is the observed probability in the ith bin of the histogram and qi is theexpected probability assuming a normal distribution around µ with standarddeviation σ. When qi = 0, that bin is ignored. The value e−|D| will reported asthe value of the aesthetic measure. With a value between 0 and 1, a low valuewill indicate a large deviation and hence a poor image, whereas a large valuewill correspond to a good image.

We justify this aesthetic measure as follows:

1. Aesthetic measure R discourages images which give rise to very high or verylow response values. If a viewer gives very little response to something, it istoo insignificant to be of interest. On the other hand, if a viewer gives a verylarge response to something, it is too disturbing or chaotic.

2. The response value increases as the gradient increases and decreases as thegradient falls. Very low gradients give rise to single coloured monotonousareas (which do not interest a viewer) whereas very large gradients give riseto sharp lines and boundaries separating areas with huge colour differences(which is undesirable). Aesthetic measure R discourages very high and verylow gradients and encourages reasonable values of gradients.

2.2 Aesthetic measure MC

This measure is based on the aesthetic theory of Machado and Cardoso [13]asserting that the aesthetic value of an artwork is directly connected to Image

Complexity (IC) and inversely connected to Processing Complexity (PC). So,the value of the aesthetic measure is calculated as the ratio

IC

PC. (1)

In order to compute IC, we first compress the image losslessly using JPEGcompression and calculate the ratio (I) of the size of compressed image to thesize of uncompressed image. We hypothesize that the IC is directly connectedto the ratio I. The inherent unpredictability, or randomness can be measuredby the extent to which it is possible to compress the data [6]. Low values of Iindicate substantially compressible and low complexity image. High values of Iindicate not very compressible and therefore more complex image. That is,

more compressible ≡ less random ≡ more predictable ≡ less complex

Hence, the less the value of ratio I (the less the size of the compressed file) is,the more compressible and hence, the less complex the image is. We substitutethe ratio I for IC in Equation 1.

PC should reflect the complexity of the coding of the image. We encode eachimage by three expression trees, one for each of the R, G and B components, asdetailed in Section 3. In order to compute PC, we compress the expression treesrepresented as strings in prefix notation and again find the ratio P of size aftercompression to size before compression. We argue that PC can be substitutedby the ratio P . The aesthetic measure MC will be computed as

I

P.

In theory the value of this aesthetic measure could range from zero to infinity,where infinity corresponds to an image that cannot be compressed, but whose ge-netic expression tree can be compressed to the minimum. Zero corresponds to animage that can be compressed significantly, but with an expression that cannotbe compressed. It is notable that the compression rate PC of the mathematicalexpressions could be replaced with the more exact rate computed for the min-imum length of an equivalent mathematical expression. However, as arithmeticsimplification is not applied on the mathematical expressions in our system, weconsider that using the actual evolved expression is appropriate.

2.3 Aesthetic measure BZ

This aesthetic measure is based on Birkhoff’s measure [2] and Zurek’s physicalentropy [17]. We compute the value of Shannon’s entropy as mentioned in [17]by creating a histogram of luminance values of pixels and computing Shannon’sentropy Hp as follows:

Hp = −∑

i

pi log pi

where pi is the probability in the ith bin of the histogram. The luminance value(L) for a pixel (i, j) is computed as follows:

L = (0.2126 ∗ ri,j) + (0.7152 ∗ gi,j) + (0.0722 ∗ bi,j).

Next, the Kolmogorov Complexity (K) [17] of the expression trees of the imageis estimated by compressing the strings corresponding to expression trees andfinding the length of the compressed string. The value of this aesthetic measureis given by

Hp

K

This aesthetic measure discourages very high and very low luminance valuesbecause it favours high values of Hp. Very high and very low luminance valueslead to low values of Hp. Here, K is used as a measure of PC.

2.4 Aesthetic measure S

As stated in [17], to analyse an image’s composition, the used measures mustquantify the degree of correlation or similarity between image parts. We computethis degree of correlation by dividing the image into four equal squares andcompute Shannon’s entropy (Hpi

, i = 1, . . . , 4) for each of these parts. We thencompute the weighted mean of these values (the weight being the area of thepart). Finally, we find the ratio of the weighted mean to the Shannon’s entropyvalue of the image as a whole to obtain the value of the aesthetic measure. Thevalue of the aesthetic measure is given by

Hp1 +Hp2 +Hp3 +Hp4

4Hp.

3 The underlying evolutionary art system

A simple interactive evolutionary art system is used, where the user is presentedwith nine images and has to select two as parents for the next generation. Imagesare represented by triplets of functions corresponding to the three components R,G and B. For each pixel of the image, the values of these functions are calculatedand produce the colour of the pixel as shown in Fig. 1. Genetic programming isemployed for the evolution of the expression trees. Simple subtree crossover, sub-tree and point mutation are the allowed genetic operators. The user can set theoperators to be used and their rates and can also introduce new random imagesat anytime during the interactive evolution. The system was implemented inJava and uses all mathematical functions provided in Java.Math. The terminalsare Cartesian and polar coordinates of the points and also random constants.3

Examples of images produced by the authors are shown in Fig. 2.

3 The interactive system is available for download at http://www.evoartmedia.com.

Fig. 1. The genetic representation used

4 Simple numerical analysis

To see whether there are any particular functions that are preferred more thanothers, we initially analysed their frequency in nice images. The number of oc-currences of each function in 44 nice images manually selected from a set ofimages generated through interactive evolution by the authors in sessions oflength varying between 15 and 30 minutes is shown in Fig. 3. The images wereselected such that on visual inspection they looked substantially different. It canbe seen that the preferred functions are SEC, CUBRT, EXP, LOG, SQRT +, -,MAX, AVG, *, as each of these occurs on average at least once in every aestheticimage. At 63.8% of all variables, polar coordinates were the preferred variablesand variables were preferred over numeric constants, as 73.3% of terminals. Theconstant range [0.5, 0.6) had the highest frequency of 20% of all constants, thenext closest being [0.6, 0.7) at 12.5%. Such a simple analysis does not really of-fer detailed understanding of human aesthetic selection, just indicates the morelikely ingredients of aesthetic images within the given class of images.

(a) Spider (b) Flower (c) Fish

Fig. 2. Example images generated by interactive evolution.

Fig. 3. Occurrences of various functions in 44 different aesthetic images obtained indifferent runs of the system.

5 Aesthetic measures to model human selection

Individual interactive evolution experiments were analysed to better understandtheir driving force: human selection. Then automatic evolution was applied tothe resulting image, using the selection criteria revealed by the analysis. Theanalysis involved monitoring the evolution of the four aesthetic measures de-scribed in Section 2 during interactive experiments performed by four differentpeople. We found that although there are similarities for all users, the occupa-tion of the user substantially influences their use of the system and their imageselection preference.4 There is no universally applicable simple model. Computerscientists tend to use the system for longer than graphic designers. The valueof measure MC shows a clear growth over generations in the case of computerscientists, as in Fig. 4(a), both for the minimum and the maximum value takenin each generation. At the same time, there is no clear tendency in the evolution4 The users were computer scientists and graphic designers.

0 5 10 15 20 25 30 35 40 450

0.2

0.4

0.6

0.8

1

1.2

1.4

Generations

Min

imum

and

max

imum

val

ue o

f mea

sure

MC

BZ

R

S

(a) Evolution of measure values

0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.30

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

2 4

6

8

10

12

14

16

1820

22

2426

28

3032

34

36

3840

42

MC

Z

(b) Pareto front (MC,BZ)

Fig. 4. Evolution of image through interactive evolution by computer scientist.

0 5 10 15 20 25 300

0.2

0.4

0.6

0.8

1

1.2

1.4

Min

imum

and

max

imum

val

ue o

f mea

sure

Generations

MCS

BZ

R

(a) Evolution of measure values

0.7 0.75 0.8 0.85 0.9 0.95 1 1.05 1.10

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

2

4

68 10

1214

1618

2022

24 2628

MC

BZ

(b) Pareto front (MC,BZ)

Fig. 5. Evolution of image through interactive evolution by graphic designer.

of values for measure MC in the case of graphic designers (see Fig. 5(a)). Theevolution of measures BZ and S are similar for both types of users: variationwithin similar ranges is observed, approximately [0, 0.05] for BZ and [0.8, 1] forS, respectively. The R measure has a lot of variation across its full range forcomputer scientists and somewhat less variation for graphic designers, but fol-lows no clear pattern. Interestingly, if we consider the evolution of two measurestogether and draw the Pareto front of non-dominated solutions5 in each gener-ation, we notice some trends. Figures 4(b) and 5(b) show the evolution of thePareto front for measures MC and BZ. In both shown examples, with a few ex-ceptions over the full experiment, the front is moving toward better (i.e. highervalues of aesthetic measures) non-dominated solutions. We interpret this as anindication that human users may not select the images during evolution in away that consistently follows a single aesthetic measure, but more likely a set ofaesthetic measures. In fact if we compare the two images in Fig. 6(a) and 6(d),we notice that the first image scores better over measures MC and BZ, while thesecond image scores better over measures R and S.

When attempting partial automatic evolution [15, 20] we propose that thehuman’s previous selections are analysed and modelled by the best fitting set ofmeasures and then the automatic evolution subsequently uses these measures.It is then more likely that images to the particular human user’s liking areproduced by automatic evolution. We therefore applied automatic evolution withthe combination of the MC and BZ fitness measures to create images startingfrom the preferred images of the human users. We experimented with mutationonly or both crossover and mutation, various settings of population sizes (15-40) and generation numbers (30-100) allowing the computer to spend differentamounts of time on creating new images. Evolved images are shown in Figures6(b) 6(c) and 6(e), 6(f), respectively.

5 A point (x1, y1) is part of the Pareto front of a set if there is no other point (x2, y2)in the same set such that x2 > x1 and y2 > y1.

R = 3.8E-33MC = 0.29BZ = 0.04S = 0.82

(a) Computer scientist (b) Automatic mut. (c) Automatic Xover

R = 0.47MC = 0.26BZ = 0.015S = 0.99

(d) Graphic designer (e) Automatic mut. (f) Automatic Xover

Fig. 6. Images created by computer scientist and graphic designer. Subsequent imagesevolved from these by automatic evolution using mutation or crossover.

6 Conclusion

We proposed modelling human user preference by a set of aesthetic measuresmonitored through observation of human selection in an interactive evolutionaryart system. Although our evolutionary art system is very simple and is only capa-ble of generating images within a limited set, it provides a suitable environmentfor studying human aesthetic judgment. The same principles could be appliedusing an extended set of aesthetic measures on more sophisticated evolutionaryart systems and then different combinations of aesthetic measures may be foundto model individual users best.

McCormack [16] criticises aesthetic selection itself and proposes an openproblem ”to devise formalized fitness functions that are capable of measuringhuman aesthetic properties of phenotypes”. The key is to model and measurehuman aesthetic properties by the available means.

We argue that once a combination of measures that models increasing hu-man preference during interactive evolution is identified, automatic evolution isprovided with a suitable fitness evaluation method. We are planning to conductmore experiments using an approach similar to that of Colton et al. [5]. Alsowe are planning to employ machine learning techniques to find potentially moreaccurate functions driving human aesthetic judgment and to subsequently applythese functions for evaluation and selection in automatic evolution.

Acknowledgments

The authors acknowledge the support of Aston University’s summer internshipscheme which made it possible for the second author to take part in the reportedwork. The authors are grateful to Philip Welch, Ken Murphy, Adrian Carolanand Daniel Baxendale for taking part in the user experiments.

References

1. S. Baluja, D. Pomerleau, and T. Jochem. Towards Automated Artificial Evolutionfor Computer-generated Images. Connection Science, 6:325–354, 1994.

2. G. D. Birkhoff. Aesthetic Measure. Harvard Univ. Press, 1933.3. S. Colton. The Painting Fool. www.thepaintingfool.com, 2007.4. S. Colton. Automatic Invention of Fitness Functions with Applications to Scene

Generation. In EvoApplications 2008, LNCS 4974, pages 381–391, 2008.5. S. Colton, J. Gow, P. Torres, and P. Cairns. Experiments in Objet Trouve Brows-

ing. In Conference on Computational Creativity, pages 238–247, 2010.6. S. Dasgupta, C. Papadimitriou, and U. Vazirani. Algorithms. Tata McGraw-Hill

Publishing Company Limited, 2008.7. R. Dawkins. The Blind Watchmaker. 1986.8. E. den Heier and A. E. Eiben. Comparing Aesthetic Measures for Evolutionary

Art. In EvoApplications 2010, Part II, LNCS 6025, pages 311–320, 2010.9. S. DiPaola and L. Gabora. Incorporating Characteristics of Human Creativity into

an Evolutionary Art System. Genetic Programming and Evolvable Machines, 2(1),2009.

10. S. Draves. Electric Sheep. electricsheep.org, 1999.11. G. Greenfield. Generative Art and Evolutionary Refinement. In EvoApplications

2010, Part II, LNCS 6025, pages 291–300, 2010.12. Y. Li and C.-J. Hu. Aesthetic Learning in an Interctive Evolutionary Art System.

In EvoApplications 2010, Part II, LNCS 6025, pages 301–310, 2010.13. P. Machado and A. Cardoso. Computing Aesthetics. In SBIA’98, LNCS 1515,

pages 219–228, 1998.14. P. Machado and A. Cardoso. All the truth about NEvAr. Applied Intelligence,

Special issue on Creative Systems, 16(2):101–119, 2002.15. P. Machado, J. Romero, A. Cardoso, and A. Santos. Partially interactive evolu-

tionary artists. New Generation computing, 23:143–155, 2005.16. J. McCormack. Open Problems in Evolutionary Music and Art. In EvoWorkshops

2005, LNCS 3449, pages 428–436, 2005.17. J. Rigau, M. Feixas, and M. Sbert. Informational Aesthetics Measures. IEEE

Computer Graphics and Applications, pages 24–34, 2008.18. B. J. Ross, W. Ralph, and H. Zong. Evolutionary Image Synthesis Using a Model

of Aesthetics. In IEEE Congress on Evolutionary Computation, pages 1087–1094,2006.

19. K. Sims. Artificial Evolution for Computer Graphics. In SIGGRAPH 1991, vol-ume 25, pages 318–328, 1991.

20. S. Todd and W. Latham. Evolutionary Art and Computers. Academic Press, 1992.21. J. J. Ventrella. Evolving the mandelbrot set to imitate figurative art. In P. F.

Hingston, C. L. Barone, and Z. Michalewicz, editors, Design by Evolution, pages145–167. 2008.