Problem Decomposition Using Indirect Reciprocity inEvolved Populations

Heather J. Goldsby, Sherri Goings, Jeff Clune, and Charles OfriaDepartment of Computer Science and Engineering

Michigan State UniversityEast Lansing, Michigan 48824 USA

hjg@msu.edu, goingssh@msu.edu, jclune@msu.edu, ofria@msu.edu

ABSTRACTEvolutionary problem decomposition techniques divide acomplex problem into simpler subproblems, evolve individ-uals to produce subcomponents that solve the subproblems,and then assemble the subcomponents to produce an over-all solution. Ideally, these techniques would automaticallydecompose the problem and dynamically assemble the sub-components to form the solution. However, although sig-nificant progress in automated problem decomposition hasbeen made, most techniques explicitly assemble the com-plete solution as part of the fitness function. In this pa-per, we propose a digital-evolution technique that lays thegroundwork for enabling individuals within the populationto dynamically decompose a problem and assemble a so-lution. Specifically, our approach evolves specialists thatproduce some subcomponents of a problem, cooperate withothers to receive different subcomponents, and then assem-ble the subcomponents to produce an overall solution. Wefirst establish that this technique is able to evolve specialiststhat cooperate. We then demonstrate that it is more effec-tive to use a generalist strategy, wherein organisms solve theentire problem themselves, on simple problems, but that aspecialist strategy is better on complex problems. Finally,we show that our technique automatically selects a gener-alist or specialist strategy based on the complexity of theproblem.

Categories and Subject DescriptorsF.1.1 [Computation by Abstract Devices]: Models ofComputation—Self-modifying machines

General TermsExperimentation.

KeywordsDigital evolution. Altruism. Binary String Cover Problem.Evolution of Cooperation.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.GECCO’09, July 8–12, 2009, Montréal, Québec, Canada.Copyright 2009 ACM 978-1-60558-325-9/09/07 ...$5.00.

1. INTRODUCTIONTraditionally, evolutionary algorithms (EAs) contain a

single population in which each individual encodes a com-plete solution to a problem. EAs work well for problemswith a search space that can be reasonably well-explored,given the computational resources available to the popula-tion. However, as the complexity of the problems addressedusing EAs increases, the size of the search space and theamount of time required to search for a satisfactory solutiongrows rapidly. Problem decomposition has been proposedto address this limitation [15]. Problem-decomposition EAsdecompose a problem into simpler subproblems, individualsproduce subcomponents (where each subcomponent is a so-lution to a subproblem), and then the subcomponents arecombined to obtain a solution to the overall problem [15].One of the overarching challenges of problem-decompositionEAs is to have the population automatically and dynami-cally both decompose the problem (if necessary) and assem-ble a complete solution.

Although significant progress has been made in problemdecomposition [3, 6, 10, 15, 17, 19], most techniques evolvespecialists that produce subcomponents and explicitly re-combine the subcomponents as part of the fitness functionto produce an overall solution [15, 19]. A notable excep-tion is the approach taken by some learning classifier sys-tems (LCSs), which evolve specialists that produce and thendynamically assemble the subcomponents to form a solu-tion [10, 17]. For example, this LCS technique been used toevolve the behavior of robots in real-world applications [4].However, LCSs are designed for rule-based problems andare thus restricted in their application. To enable problem-decomposition EAs to solve complex, non-rule-based prob-lems, we must explore other techniques for dynamically de-composing a problem and assembling a solution.

In this paper, we propose a technique for problem decom-position that lays the groundwork for evolving specialiststhat automatically decompose a problem, produce subcom-ponents, and then dynamically assemble the subcomponentsto form a complete solution. Specifically, our technique cur-rently evolves specialists that produce subcomponents andcooperate with other neighboring specialists to collect all ofthe subcomponents necessary to form a complete solution.Because an individual organism collects all of the subcom-ponents required for the complete solution, this approachhas the potential to be used to address problems that re-quire more complex assembly procedures, which can also beevolved.

105

We implement our approach in Avida [14], a digital-evolution platform previously used to study the origin ofcomplex features [9], the evolutionary design of modular-ity [12], evolutionary robustness [8, 18], and the evolution ofaltruism [1, 5]. Within an Avida experiment, a populationof self-replicating computer programs exists in a user-definedcomputational environment and is subject to mutations andnatural selection. These“digital organisms”compete for lim-ited resources and are subjected to mutations.

For this study, we provide instructions and infrastructureto allow organisms to cooperate in order to obtain subcom-ponents produced by neighboring organisms using indirectreciprocity [11]. Whereas direct reciprocity involves the ex-change of altruistic donations during repeated interactionsbetween the same two individuals, indirect reciprocity in-volves the exchange of donations for reputation, where fu-ture donations are directed toward individuals with a highreputation [11, 16]. In this study, an organism’s reputa-tion is an accurate reflection of its behavior that is auto-matically updated. Specifically, an organism can donate asubcomponent to a neighboring organism. As a result ofthis donation, the donor organism’s reputation automati-cally increases. This improved reputation, in turn, increasesthe probability that the donor organism will subsequentlyreceive donations of subcomponents from others. Thus, anorganism can assemble a complete solution by combining thesubcomponents it has produced itself with subcomponentsthat it has received from others.

We demonstrate the potential for this approach using avariant of the binary string cover problem presented in [15].Whereas the original binary string cover problem considersa set of individuals to be a solution, we explicitly requireindividuals to cooperate to assemble a complete solution. Inthis case, assembling a solution simply requires an organismto collect all of the subcomponents. First, to test whetherthis approach can evolve cooperation, we force individualsto be specialists that produce only one subcomponent andcooperate to solve the complete problem. Next we comparethe performance of a population of generalists that each at-tempt to solve the entire problem to the performance of apopulation of specialists. On simpler problems, the popula-tion of generalists outcompete the population of specialists.However, as the complexity of the problem increases, thepopulation of specialists outperforms the population of gen-eralists. Lastly, we demonstrate that when allowed to beeither generalists or specialists, the population compositionautomatically changes to solve the problem most efficiently.Specifically, we enable both generalists and specialists tocoexist within a population. We run experiments of vary-ing complexity and show that the population compositionchanges from consisting of primarily generalists to primarilyspecialists as the problem complexity increases.

2. RELATED WORKNumerous techniques have been proposed to automate the

evolution of problem decomposition [3, 6, 10, 15, 17, 19].In general, these techniques evolve specialists that producesubcomponents. The overall solution is considered to bea group of specialists comprising either one representativemember of each species (e.g., [6, 15, 19]) or an EA-selectedteam (e.g., [3, 10, 17]).

Several approaches assemble the subcomponents as partof the fitness function [15, 19]. Specifically, the cooperative

coevolution architecture [15] evolves two or more species incompletely isolated populations. Cooperation between thesespecies occurs at the time of fitness evaluation, when indi-viduals from one species are evaluated with representativesfrom each of the other species. The nature of these col-laborations is determined by the user, as is the choice ofrepresentatives from each species, but the number of speciesis determined by the algorithm automatically. Additionally,Yong and Miikkulainen [19] apply the cooperative coevolu-tion architecture to evolve coordinated predator behavior ina prey-capture task. Three Enforced SubPopulations [7] arecreated, each of which evolves an artificial neural networkthat represents the behavior for one predator [19]. Duringthe fitness evaluation, a representative neural network fromeach subpopulation is placed in a simulated environmentand the reward for captured prey is split equally amongall representatives present. Unfortunately, these approachesare only applicable in problems where the general form ofbetween-species collaborations is known a priori.

Other approaches enable the EA itself to dynamically as-semble the subcomponents. These approaches are typicallybased on a market economy. For example, some learningclassifier systems form teams of individuals that bid for theirsubcomponents to be used as part of the solution to theclassification problem [10, 17]. Additionally, Cornforth andKirley [3] propose a non-rule based approach to problemdecomposition using a market-based model, where agentsgroup together in a hierarchical fashion to form complexproblem solutions. One limitation of this approach is thatindividuals do not dynamically evolve subcomponents of theproblem, but rather innately represent preformed subcom-ponents.

3. AVIDAFigure 1 depicts an Avida population and the structure

of an individual organism. Each digital organism consistsof a circular list of instructions (its genome) and a virtualCPU that executes those instructions. The standard Avidainstruction set is Turing complete and is designed so thatrandom mutations will always yield a syntactically correctprogram, albeit one that may not perform any meaningfulcomputation [13]. Within their virtual environment, organ-isms can produce and consume resources, aid neighboringorganisms through altruistic donations, and sense or changevarious properties of the environment.

An Avida population comprises a number of cells, wherea cell is a compartment in which an organism can live. Eachcell can contain at most one organism, and the size of anAvida population is bounded by the number of cells in theenvironment. Cells are connected to each other via a con-figurable topology, generally a grid or torus, that definesthe neighborhood of each cell. An organism’s neighbors arethe organisms that live in the cells adjoining its own cell.Additionally, each organism has a facing, which is one con-nection to a neighboring cell. Organisms are able to changetheir facing by executing rotate instructions.

Organisms are self-replicating meaning that the genomeitself must contain the instructions to create an offspring.When an organism replicates, a cell to contain the offspringis selected from the environment at random, and any pre-vious inhabitant of the target cell is replaced (killed andoverwritten) by the offspring. Each population starts withone or more organisms that are capable only of replication,

106

and different genomes are produced through random muta-tions introduced during replication. Mutation types include:replacing the instruction with a different one, inserting anadditional, random instruction into the offspring’s genome,and removing an instruction from the offspring’s genome.

organismLegend

k shift-r

s h-alloc

a nop-Ay get-heado adde if-less

e if-less

m incC rotate-r

D send-msg

p sub

y get-head

u h-copy

p sub

u h-copy

cell

q nandh push

z if-labelu h-copy

b nop-B

D send-msg

A set-flow

z if-labelu h-copy

z if-labelF get-idc nop-CD send-msg

u h-copy

E retrieve-msgF get-idf if-grtw mov-head

v h-searchi swap-stk

t h-divide

C rotate-r

send-msg

Flow-control CellInstruction

CPU

ReadWrite

Heads

AXBXCX

Registers

GSLS

Stacks

Interface

Figure 1: The AVIDA platform

Organisms can perform tasks that allow them to metab-olize resources from their environment. Metabolizing ben-eficial resources increases an organism’s merit, which de-termines the rate its virtual CPU will execute instructionsrelative to the other organisms in the population. For exam-ple, an organism with a merit of 2 will, on average, executetwice as many instructions as an organism with a merit of1. Since digital organisms are self-replicating and competefor space, a higher merit (all else being equal) results in anorganism that replicates more rapidly, spreading throughoutand eventually dominating the population.

The amount of merit that an organism gains for complet-ing a task depends on the type and abundance of resourcesassociated with the task. These resources may be unlimited,such that they are always available and the merit increasegained for completing them is effectively a user-defined con-stant. Or the resources may be limited, such that the meritincrease gained is dependent on the amount of the associatedresource that is currently available. A small amount of eachlimited resource continually flows into the environment to re-plenish the resource stores. When many organisms executea task associated with a limited resource, the amount of re-source available decreases until additional organisms wouldreceive more merit by completing a task associated with adifferent resource. Cooper and Ofria have shown that usinglimited resources in Avida leads to greater diversity and thatwith them multiple species can stably coexist in an asexualpopulation [2]. In this paper, we use tasks associated withboth limited and unlimited resources.

4. EXPERIMENTAL SETUPIn this section, we provide an overview of the binary string

production problem that we use to illustrate our technique,discuss the modifications to Avida that were necessary toallow specialists to cooperate, and provide the standard con-figurations used for our experiments.

4.1 Binary String Production ProblemThe original binary string cover problem, which we are

using a variation of, considers each individual in a popula-tion to represent a binary string x. The objective of eachindividual is to match as strongly as possible a set V of Nbinary vectors called strings, where each string vi in V is oflength l. Let x be a match value, where bi is the number ofbits exactly matched (value and placement) between x andvi. Formally, x is defined as follows:

• If bi < l2, then S(x, vi) = 0.

• If bi ≥ l2, then S(x, vi) = ( 2bi

l− 1)2.

Unless the strings in set V are identical, no single value ofx will be able to perfectly optimize all objectives, and thustrade-offs must occur. The EA’s solution to the binary stringcover problem is typically considered to be a set of individ-uals in the final population [6, 15]. If individuals within thepopulation specialize, then they are able to produce a betteroverall solution. Goings and Ofria have previously demon-strated the ability of Avida to develop specialists that covermultiple niches in this problem [6].

Whereas the original binary string cover problem stati-cally composes subcomponents by considering a set of in-dividuals (one from each species) to represent the solution,this variation increases the difficulty of the original problemby requiring each individual to produce or acquire all of thestrings and assemble the complete solution. We refer to thisvariation as the binary string production problem. Thus, theobjective of the individual is, given a set V of binary strings,to produce multiple exact copies of V . We refer to a perfectcopy of V as a complete set.

For our experiments, we use several variants of the bi-nary string production problem, where each variant differsin complexity. Specifically, we define 4-bit, 8-bit, 16-bit, and24-bit variants. Each variant has two strings that constituteV : the first string is all zeros and the second string is allones. For example, the 4-bit variant uses strings 0000 and1111 and the complete set is {0000, 1111}.

4.2 Avida ExtensionsTo enable organisms to specialize and cooperate to pro-

duce complete solutions, we extended the organism infras-tructure, defined instructions to allow organisms to producecomplete sets, and defined tasks that described the desiredoutputs. At a high level, an organism creates a string in itsbuffer and then produces copies of the string. The copiescan be donated to increase its reputation or used as partsof a complete set. Tasks are associated with the quality ofstrings produced, quantity of strings produced, and also thequantity of complete sets produced.

4.2.1 Organism InfrastructureFigure 2 (a) depicts the four key elements of an Avida

organism that are used to produce solutions to the binarystring production problem. First, an organism has a bufferin which it is able to construct strings. The buffer is ini-tially empty and is exactly the length of a string. A pointerinto the buffer determines where insertion will occur. Thepointer starts at the first element, and moves one positioneach time a bit is inserted into the buffer. After the lastelement has been written, the pointer returns to the firstelement and future insertions overwrite existing bits. Forexample, Figure 2 (a) depicts a buffer of length 4 (for the4-bit variant of the string production problem). Second, an

107

organism has two counters that track the number of copiesof each string that the organism has at its disposal, where acopy could be either produced by the organism or receivedas a donation. For example, Figure 2 (a) depicts the twocounters (c0000 and c1111). The number of copies of a stringis limited to a user-defined amount. For these experiments,the limit is 10 copies per string. A copy can be used as partof a complete set or as a donation. Third, an organism hasa tag that identifies which vector it is best at producing.An organism initially inherits the tag of its parent. As theorganism replicates, its tag and the tag of its offspring areupdated to reflect the vector it is best at producing. Lastly,an organism has a reputation that is automatically updatedas a result of the organism’s cooperative actions. An organ-ism’s reputation is determined using a variant of a standingstrategy [16] as follows 1:

• reputation = 0: An organism has neither donated astring nor received a string. All organisms are born inthis state.

• reputation = -1: An organism has not donated astring, but has received a string.

• reputation = 1 An organism has donated a stringand may or may not have received a string.

4.2.2 InstructionsTable 1 depicts the instructions we added to the stan-

dard Avida instruction set [13]. Instructions insert-0 andinsert-1 insert a 0 or a 1 into the organism’s buffer. Figure 2(b) depicts an organism after it executed insert-0. Instruc-tion prod-string reads an organism’s buffer. If the string inthe buffer perfectly matches one of the vectors in V , theorganism’s counter for this vector is incremented by 1; oth-erwise, the instruction has no effect. For example, Figure 2(c) depicts the buffer of an organism that has constructedstring 0000 and then executed prod-string. The counter forthe string 0000, c0000 is incremented. The buffer remains thesame after the prod-string instruction is executed. Thus, anorganism may consecutively execute the instruction severaltimes to create multiple copies of the same string.

The remainder of the new instructions allow organisms tocooperate through indirect reciprocity. To donate a string,an organism executes the instruction donate-string, whichdecrements the organism’s counter for the string and incre-ments the corresponding counter of the neighbor. An organ-ism’s tag determines which string it donates. For example, ifan organism’s tag is 0000, then the organism donates string0000. Donations between organisms of the same tag wouldnot assist organisms in constructing a complete set and thusare not allowed. Additionally, if an organism does not haveany copies of the string it donates (e.g., c0000 = 0), then thedonation does not occur and the counters and reputationsof the organism and its neighbor remain unchanged.

Organisms are able to sense their own reputation and thereputation of their facing neighbor using the get-reputationand get-neighbors-reputation instructions, respectively. Whenthese instructions are executed, the reputation value isplaced in one of the organism’s registers. Lastly, the rotate-to-rep-tag instruction rotates an organism to face the neighborwith the highest reputation of the opposite tag. If the or-

1We also experimented with reputation strategies in whichan organism’s reputation increased with each subsequent do-nation. These strategies did not qualitatively change ourresults.

c : 0

c : 00000

1111

reputation: 0

tag: 0000

c : 0

c : 00000

1111

buffer

0

pointer(b) An organism’s

internal state

after inserting 0.

(c) An organism’s

internal state

after executing

prod−string.reputation: 0

tag: 0000

c : 0

c : 10000

1111

(a) An organism’s

initial state.

buffer

pointer

buffer

0 0 0 0

pointer

reputation: 0

tag: 0000

Figure 2: The infrastructure that allows Avida or-ganisms to produce strings, donate strings, and co-operate using indirect reciprocity.

ganism does not have a neighbor of the opposite tag, thenits facing remains unchanged. If multiple neighboring or-ganisms of equally high reputation all have the opposite tag,then the organism is rotated to face a random organism fromthis set.

Table 1: Additional Avida instructions that alloworganisms to produce strings, donate strings, andselectively cooperate with neighbors on the basis ofreputation.

Instruction Descriptioninsert-0 Insert 0 into the bufferinsert-1 Insert 1 into the bufferprod-string If the string in the buffer is a vector

vi, increment the counter of vi

donate-string Donate a copy of a string to the fac-ing neighbor

get-reputation Get the organism’s reputationget-neighbors-reputation Get the neighbor’s reputationrotate-to-rep-tag Rotate to the neighbor with a differ-

ent tag and the highest reputation

4.2.3 TasksWe defined two types of tasks associated with resources

that reward organisms for the binary string production prob-lem. The first task, MatchString, is associated with a limitedresource and is used to ensure that the population main-tains the ability to produce both strings. The performanceof an organism on MatchString is proportional to the qual-ity of the string in its buffer when the organism replicates.A MatchString task is defined for both strings in V (e.g.,MatchString0000 and MatchString1111) and the value of the taskis defined using the same formula as the original bit stringcover problem. Additionally, if the organism has producedthe string that perfectly matches the vector itself, then itreceives a score of 1. The formula is:

• If bi < l2, then MatchString(org, vi) = 0.

• If bi ≥ l2, then MatchString(org, vi) = ( 2bi

l− 1)2.

• If cvi ≥ 1, then MatchString(org, vi) = 1.

108

The MatchString task also labels the organism with a tagthat corresponds to the vector it is best at producing. Specif-ically, an organism’s tag corresponds to the vector whoseMatchString task had the highest quality.

The second type of task, CompleteSet, is associated withan unlimited resource that rewards organisms for produc-ing complete sets. Specifically, the task quality is equal tofour times the number of complete sets plus the number ofadditional copies of either string. Task quality is partiallydefined based on strings that cannot be used as part of acomplete set to reward organisms for production quantity.Formally, let clow be the counter with the lowest balance,ballow. Let chigh be the counter with the higher balance,balhigh. The formula for computing the reward of the Com-pleteSet function is:

CompleteSet(org) = (4 ∗ ballow) + (balhigh − ballow)

4.3 Experimental SetupOur experiments all used a common set of configurations.

Each experiment comprised 20 replicate runs to account forthe stochastic nature of the evolutionary process. Each runhad 3,600 cells (and thus a maximum population size of3,600 organisms) arranged in a toroidal topology. Each runstarted with two untagged asexual self-replicating organismsthat constructed a string of the appropriate length for thevariant of the problem being attempted. One organism con-structed a random string and the other constructed the com-plement of the random string. Seeding the organisms withstrings assists the organisms in solving the bit string pro-duction problem because, to receive the resources associatedwith completing a MatchString task, an organism must con-struct a string that has over 50% of the bits correct.

The starting length of the organisms was 100 instructions.When an organism replicated, each instruction had a 0.0075probability of mutating as it was copied. Additionally, eachgenome had a 0.05 probability of an insertion mutation anda 0.05 probability of a deletion mutation. The offspring of anorganism was placed in a random location in the torus. Thismass action replacement strategy ensures that neighboringorganisms are unrelated and thus cooperation would not besimply because of kin selection [1, 5]. The runs lasted for50,000 updates, where an update is the standard unit oftime in Avida and corresponds to the time required for eachorganism to execute, on average, 30 instructions.

5. EXPERIMENTAL RESULTSIn this section, we demonstrate our problem decomposi-

tion technique on the bit string production problem. First,to ensure that organisms are able to dynamically cooper-ate to solve the problem, we require all organisms to bespecialists and exploit the information produced by otherorganisms to cooperate. Because the specialist strategy in-curs the overhead of cooperation, a generalist strategy mayoutperform the specialist strategy on simple problems. Totest this, we compare the performance of a population ofspecialists and a population of generalists on the bit stringproduction problem. We then enable organisms to be gener-alists or specialists and demonstrate that Avida varies thepopulation composition in response to the complexity of theproblem.

5.1 Evolving SpecialistsOur first set of experiments test whether Avida can evolve

a population of specialists that cooperate to solve binarystring production problems of varying complexity (i.e., 4-bit, 8-bit, 16-bit, and 24-bit strings). We force all organismsto be strict specialists, meaning that they produce stringsthat match at most one string and therefore must cooperateto create complete sets. To ensure all organisms are strictspecialists, we modified the prod-string instruction so thatan organism can produce copies only of the string that itis tagged with. Thus, if an organism is tagged with string0000, then it can only produce copies of string 0000 andmust receive string 1111 as a donation.

Although we provide the instructions necessary for theorganisms to cooperate using indirect reciprocity, the pop-ulation must evolve an algorithm that effectively uses theseinstructions to produce complete sets. It is possible that apopulation could cooperate using a different technique. Toisolate the critical information necessary to enable cooper-ation among unrelated specialists, we ran four treatmentsthat varied the ability of the organisms to sense tags andreputations. Specifically, the four treatments are:

• Organisms do not have the ability to sense either tagsor reputation. (no rep, no tags)

• Organisms have the ability to sense tags, but not theability to sense reputation. (no rep, tags)

• Organisms have the ability to sense reputation, butnot the ability to sense tags. (rep, no tags)

• Organisms have the ability to sense both reputationand tags. (rep, tags)

We ran 20 replicates for each variant of the binary stringproduction problem and evaluated the results according tothe number of complete sets produced and the number oforganisms that produced complete sets. The relative rank-ing of the treatments was the same for all variants. Figure 3depicts the results of varying the ability of the organismsto sense tags and reputations for the 16-bit variant. Eachtreatment produced significantly different results (p < 0.05,Mann-Whitney U test). Specifically, Figure 3 (a) depicts thenumber of organisms that produced complete sets and Fig-ure 3 (b) depicts the number of complete sets produced bythe organisms. The solid lines are the mean of 20 treatmentsand the dotted lines are one standard error of the mean.Overall, organisms without either tag or reputation informa-tion (green line with triangles) performed poorly; whereas,organisms with both tag and reputation information (blueline with circles) achieved both the largest number of organ-isms producing complete sets(∼2,000) and also the largestnumber of complete sets produced (∼10,000).

Counterintuitively, organisms with reputation informa-tion, but not tag information (black line with upside downtriangles) produced fewer complete sets than organismswithout any information. We performed further investiga-tions and found out that, although these organisms are pro-ducing fewer complete sets, their average fitness exceeds thatof organisms without any information. Essentially, their fit-ness is greater because they are producing more copies of asingle string. Additionally, organisms with tag information,but not reputation information (red line with squares) per-formed slightly better than organisms with no information,but not nearly as well as organisms with both reputationand tag information (p < 0.05, Mann-Whitney U test).

109

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 104

0

500

1000

1500

2000

Update

Org

an

ism

s c

rea

tin

g c

om

ple

te s

ets

No rep, no tags

No rep, tags

Rep, no tags

Rep, tags

(a) Number of organisms that produce complete sets

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 104

0

2000

4000

6000

8000

10000

Update

Com

ple

te S

ets

(b) Number of complete sets produced

Figure 3: Number of organisms that produce com-plete sets (top) and the number of complete setsproduced (bottom) with four different treatmentsthat vary the tagging and reputation informationavailable for the 16-bit variant. Populations whereorganisms have both sources of information signifi-cantly outperform populations that lack either.

Based on these experiments, we conclude that the spe-cialist strategy is affected by the organism’s ability to senseboth tags and reputation. For the remainder of the exper-iments that we describe in this paper, organisms can senseboth reputation and tag information.

5.2 Comparing Isolated Populations ofGeneralists and Specialists

The primary motivation for using a problem decomposi-tion approach is to address problems of increased complex-ity that are difficult for a standard EA to address. Next, wecompare the performance of strict specialists to strict gen-eralists, where strict generalists do not have a donate-stringinstruction and thus are unable to cooperate. To create acomplete set, a strict generalist must first create one string(e.g., 0000) in its buffer, produce copies of the string, createthe other string in its buffer (e.g., 1111), and then producecopies of it.

Our dual hypotheses for these experiments are that:1. If a problem is simple, then a population of general-

ists will outperform a population of specialists, sinceit is easier to produce both strings than to cooperate.Thus, for simpler variants of the problem, the general-ists treatments should both have more organisms that

produce complete sets and have more complete setsproduced than the specialist experiments.

2. If the problem is complex, then the specialists will out-perform the generalists, since it is easier to cooperatethan to produce both strings. Thus, the specialist ex-periments should both have more organisms that pro-duce complete sets and have more complete sets pro-duced than the generalist experiments for the morecomplex variants of the bit string production problem.

We tested these hypotheses by running 20 replicates ofstrict generalists and 20 replicates of strict specialists foreach of the four variants of the bit string production prob-lem. The results of the experiments are depicted in Figure 4,where the red lines with squares represent the generalistsand the blue lines with circles represent the specialists. Forthe simpler 4-bit and 8-bit variants, the generalists outper-form the specialists both in terms of the number of organ-isms producing complete sets and also in terms of the num-ber of complete sets produced (p < 0.05, Mann-Whitney Utest). However, for the more complex 16-bit and 24-bit vari-ants, the specialists outperform the generalists (p < 0.05,Mann-Whitney U test). The performance of the special-ists slightly increases with the complexity of the problem.Specifically, approximately the same number of specialistorganisms (∼2000) produced approximately the same num-ber of complete sets (∼10,000) for the 8-bit, 16-bit, and24-bit treatments. In contrast, the success of the generalistswas dependent upon the complexity of the problem. Forthe 4-bit and 8-bit treatments, ∼2,400 generalist organismsproduced ∼20,000 complete sets, but for the 16-bit and 24-bit treatments, ∼400 generalists produced ∼4,000 completesets. These results support our hypotheses that complexproblems favor specialists, whereas simpler problems favorgeneralists.

5.3 Mixed Populations ofGeneralists and Specialists

Ideally, a problem decomposition approach would evolvegeneralists or specialists depending on the complexity of theproblem. To test the ability of our technique to automati-cally vary population composition, we enable both types oforganisms to coexist within a population.

Our hypothesis for the next set of experiments is: Givena population in which organisms could be either generalistsor specialists, the composition of the population will changedepending on the complexity of the problem. Specifically, ifthe problem is simpler, then the generalists should dominate;if the problem is more complex, then the specialists shoulddominate. However, enabling organisms to be either gener-alists or specialists should not negatively affect the abilityof the population to solve the problem.

To test this hypothesis, we ran 20 replicates for each of thebit string production problem variants in which organismscould be either generalists or specialists. Specifically, weboth enabled the organisms to donate and did not limit themto producing one string. For these results, we consider anorganism to be a specialist if it produces only one string andto be a generalist if it produces both.

The green line with triangles in Figure 4 depicts the re-sults of a mixed population of generalists or specialists. Ingeneral, the performance of the mixed population towardthe end of the run is equivalent to that of the isolated treat-ment (either generalist or specialist) that performs the best.

110

(b)

8 bit strings 16 bit strings 24 bit strings

4 bit strings 8 bit strings 16 bit strings 24 bit strings

0 1 2 3 4 5

x 104

0

1000

2000

3000

Update0 1 2 3 4 5

x 104

0

1000

2000

3000

Update0 1 2 3 4 5

x 104

0

1000

2000

3000

Update0 1 2 3 4 5

x 104

0

1000

2000

3000

Update

0 1 2 3 4 5

x 104

0

0.5

1

1.5

2

2.5x 10

4

Update0 1 2 3 4 5

x 104

0

0.5

1

1.5

2

2.5x 10

4

Update0 1 2 3 4 5

x 104

0

0.5

1

1.5

2

2.5x 10

4

Update0 1 2 3 4 5

x 104

0

0.5

1

1.5

2

2.5x 10

4

Update

pro

ducin

g c

om

ple

te s

ets

Num

ber

of org

anis

ms

(a)

pro

duced

Num

ber

of com

ple

te s

ets

4 bit strings

Figure 4: The number of organisms producing complete sets and the number of complete sets produced bygeneralists, specialists, and mixed populations of both generalists and specialists across the four bit stringproduction problem variants. For simpler problems, generalists outperform specialists; whereas, for morecomplex problems, specialists outperform generalists. Mixed populations tend to perform at least as well asthe isolated populations of generalists or specialists.

Perc

ent of org

anis

ms

8 bit strings 16 bit strings 24 bit strings

0 1 2 3 4 5

x 104

0

20

40

60

80

100

Update0 1 2 3 4 5

x 104

0

20

40

60

80

100

Update0 1 2 3 4 5

x 104

0

20

40

60

80

100

Update0 1 2 3 4 5

x 104

0

20

40

60

80

100

Update

4 bit strings

Figure 5: The population composition of mixed populations comprising both generalists and specialists for thefour variants of the bit string production problem. In general, the more simple variants have a predominantlygeneralist population and the more complex variants have a predominantly specialist population.

Additionally, the number of complete sets produced by themixed populations for the 16-bit and 24-bit variants is higherthan that of the isolated populations of both specialists andgeneralists (p < 0.05, Mann-Whitney U test). Therefore,we conclude that the mixed population does not negativelyaffect Avida’s ability to solve the problem.

The average population compositions from the four vari-ants are depicted in Figure 5, where the red lines withsquares represent the percent of organisms that are gen-eralists and the blue lines with circles represent the per-cent of organisms that are specialists. As the complexity ofthe problem increases, so does the prevalence of specialists.Specifically, in the 4-bit and 8-bit variants, generalists tendto dominate the population (p < 0.05, Mann-Whitney Utest). However, in the 16-bit and 24-bit variants, the spe-cialists tend to dominate the population (p < 0.05, Mann-Whitney U test).

Although the average population compositions make itappear as if the number of generalists in a population grad-ually increases, for the individual replicates, once a general-ist strategy for producing complete sets is found, it quicklysweeps the population. Thus, in many cases, a specialist

strategy tended to thrive early on, probably because it issimpler to evolve, but was quickly replaced when a general-ist strategy evolved. Three possible factors that could influ-ence the ability of generalist strategy to sweep the popula-tion are: First, because generalists do not rely on others fordonations of subcomponents, they are a more robust strat-egy. Second, generalist organisms are, most likely, stingyin the sense that they do not donate. Figure 6 depicts thepercent of altruists (organisms that donate a string) in thepopulation, the percent of specialists, and the percent ofgeneralists for the 16-bit variant with a mixed population.The percent of altruists almost perfectly tracks the percentof specialists, but appears unrelated to the percent of gener-alists. Because generalists do not donate, as the number ofgeneralists increases, it becomes increasingly challenging forthe specialists to find neighboring organisms with which tocooperate. Third, generalists likely benefit from some dona-tions from specialists, since a generalist is indistinguishablefrom a specialist that has not yet donated in that both havea reputation of 0.

111

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 104

0

20

40

60

80

100

Update

Perc

ent of org

anis

ms

Generalists Specialists Altruists

Figure 6: The percent of organisms that are gen-eralists, specialists, and altruists. The percent ofaltruists in the population closely tracks the num-ber of specialists in the population, indicating thatgeneralists are not making donations.

6. CONCLUSIONS AND FUTURE WORKIn this paper, we have presented a digital-evolution based

approach to problem decomposition. Our approach enablesindividuals to be either generalists that independently solvethe problem or specialists that each contribute to solvingthe overall problem. We have demonstrated the feasibilityof this technique using several variants of the binary stringproduction problem. As the complexity of the problem in-creases, the population composition automatically changesfrom predominantly generalists to predominately specialists.However, once a generalist strategy evolves within the pop-ulation, it quickly dominates.

Our next step in this work is to increase the complex-ity of the problem by adding additional strings or longerstrings. Additionally, we will apply our approach to prob-lems that require complicated steps to assemble the overallsolution, such as with complex arithmetic formulae. To ap-ply this approach to arithmetic problems, we will expandthe Avida infrastructure to enable organisms to specializein performing a mathematical operation (e.g., addition) andto donate the results of the operation (e.g., the sum). Anorganism’s reputation will be used to indicate both the qual-ity and quantity of the results it donates. An organism’s tagwill reflect the operation it has specialized in. Finally, weare investigating the effect of changing from the explicitlydesigned reputation framework we provided to an evolvedreputation and tag system.

7. REFERENCES[1] J. Clune, H. J. Goldsby, C. Ofria, and R. T. Pennock.

Digital evolution confirms and informs the inclusivefitness theory prediction that selection favorsincreasingly accurate altruism targeting mechanisms.In preparation.

[2] T. Cooper and C. Ofria. Evolution of stableecosystems in populations of digital organisms. InEighth International Conference on Artificial Life,pages 227–232, Sydney, Australia, 2002.

[3] D. Cornforth and M. Kirley. Cooperative problemsolving using an agent-based market. In Genetic and

evolutionary computation conference (GECCO), pages60–71, Seattle, Washington, 2004.

[4] M. Dorigo and M. Colombetti. Robot Shaping: AnExperiment in Behavior Engineering. MITPress/Bradford Books, 1998.

[5] S. Goings, J. Clune, C. Ofria, and R. T. Pennock.Kin-selection: The rise and fall of kin-cheaters. InNinth International Conference on Artificial Life,pages 303–308, Boston, MA, 2004.

[6] S. Goings and C. Ofria. Ecological approaches todiversity maintenance in evolutionary algorithms. InIEEE Symposium on Artificial Life, Nashville, TN,2009.

[7] F. J. Gomez and R. Miikkulainen. Solvingnon-markovian control tasks with neuroevolution. InIn Proceedings of the 16th International JointConference on Artificial Intelligence, pages 1356–1361,1999.

[8] R. E. Lenski, C. Ofria, T. C. Collier, and C. Adami.Genome complexity, robustness, and geneticinteractions in digital organisms. In Nature, volume400, pages 661–664, 1999.

[9] R. E. Lenski, C. Ofria, R. T. Pennock, and C. Adami.The evolutionary origin of complex features. InNature, volume 423, pages 139–144, 2003.

[10] P. Lichodzijewski and M. I. Heywood. Managingteam-based problem solving with symbiotic bid-basedgenetic programming. In Genetic and evolutionarycomputation conference (GECCO), pages 363–370,Atlanta, Georgia, 2008.

[11] M. A. Nowak. Five rules for the evolution ofcooperation. Science, 314(5805):1560–1563, December2006.

[12] C. Ofria and C. Adami. Evolution of geneticorganization in digital organisms. In Proc. of DIMACSworkshop Evolution as Computation, pages 167–175,Princeton, NJ, 1999.

[13] C. Ofria, C. Adami, and T. C. Collier. Design ofevolvable computer languages. IEEE Transactions inEvolutionary Computation, 6:420–424, 2002.

[14] C. Ofria and C. O. Wilke. Avida: A software platformfor research in computational evolutionary biology.Journal of Artificial Life, 10:191–229, 2004.

[15] M. A. Potter and K. A. DeJong. Cooperativecoevolution: An architecture for evolving coadaptedsubcomponents. Evol. Comput., 8(1):1–29, 2000.

[16] R. Sugden. The economics of rights, co-operation andwelfare. Oxford: Basil Blackwell, 1986.

[17] R. Sun and D. Qi. Marlbs: Team cooperation throughbidding. In International Journal of ComputationalIntelligence Research, 2005.

[18] C. O. Wilke, J. L. Wang, C. Ofria, C. Adami, andR. E. Lenski. Evolution of digital organisms at highmutation rate leads to survival of the flattest. InNature, volume 412, pages 331–333, 2001.

[19] C. H. Yong and R. Miikkulainen. Cooperativecoevolution of multi-agent systems. Technical report,University of Texas at Austin, 2001.

112