population structure and coalescence in pedigrees ... · to construct gene genealogies, the...

Population structure and coalescence in pedigrees: comparisons to the structuredcoalescent and a framework for inference

Peter R. Wiltona,∗, Pierre Baduela,c, Matthieu M. Landonb,c, John Wakeleya

aDepartment of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA, 02138, USAbDepartment of Systems Biology, Harvard University, Cambridge, MA, 02138, USA

cEcole des Mines de Paris, Mines ParisTech, Paris 75272, France

Abstract

Contrary to what is often assumed in population genetics, independently segregating loci do not have completelyindependent ancestries, since all loci are inherited through a single, shared population pedigree. Previous work hasshown that the non-independence between gene genealogies of independently segregating loci created by the populationpedigree is weak in panmictic populations, and predictions made from standard coalescent theory are accurate forpopulations that are at least moderately sized. Here, we investigate patterns of coalescence in pedigrees of structuredpopulations. We find that population structure creates deviations away from the predictions of standard theory thatpersist on a longer timescale than in panmictic populations. Nevertheless, we find that the structured coalescent providesa reasonable approximation for the coalescent process in structured population pedigrees so long as migration events aremoderately frequent and there is no admixture in the recent pedigree of the sample. When the sampled individuals haveadmixed ancestry, we find that distributions of coalescence in the sample can be modeled as a mixture of distributionsfrom different sample configurations. We use this observation to motivate a maximum-likelihood approach for inferringmigration rates and population sizes jointly with features of the recent pedigree such as admixture and relatedness.Using simulation, we demonstrate that our inference framework accurately recovers long-term migration rates in thepresence of recent admixture in the sample pedigree.

Keywords: pedigree, coalescent theory, mixture distribution, identity-by-descent, demographic inference

1. Introduction

The coalescent is a stochastic process that describes howto construct gene genealogies, the tree-like structures thatdescribe relationships among the sampled copies of a gene.Since its introduction (Kingman, 1982a,b; Hudson, 1983;Tajima, 1983), the coalescent has been extended and ap-plied to numerous areas in population genetics and is nowone of the foremost mathematical tools for modeling ge-netic variation in samples (Hein et al., 2005; Wakeley,2009).

In a typical application to data from diploid sexual or-ganisms, the coalescent is applied to multiple loci thatare assumed to have independent ancestries because theyare found on different chromosomes and thus segregateindependently, or are far enough apart along a singlechromosome that they effectively segregate independently.Even chromosome-scale coalescent-based inference meth-ods that account for linkage and recombination (e.g., Liand Durbin, 2011; Sheehan et al., 2013; Schiffels andDurbin, 2014) multiply probabilities across distinct chro-mosomes that are assumed to have completely independenthistories due to their independent segregation.

∗Corresponding authorEmail address: [email protected] (Peter R. Wilton)

In reality, the ancestries of independently segregatingloci are independent only after conditioning on the popu-lation pedigree, which is the set of relationships betweenall individuals in the population throughout all time. Thepredictions of standard coalescent theory implicitly aver-age over the outcome of reproduction and thus over pedi-grees. Since all genetic material is inherited through asingle, shared population pedigree, applying these proba-bilities to multiple loci sampled from a single populationignores the non-independence between unlinked loci in-duced by the population pedigree.

This non-independence was examined by Wakeleyet al. (2012), who studied gene genealogies of loci seg-regating independently through pedigrees of diploid pop-ulations generated under basic Wright-Fisher-like repro-ductive dynamics, i.e., populations with constant popu-lation size, random mating, non-overlapping generations,and lacking population structure. In this context, it wasfound that the shape of the population pedigree affectedcoalescence probabilities mostly in the first ∼ log2(N) gen-erations back in time, where N is the population size, andthat in general it was difficult to distinguish distributionsof coalescence times that were generated by segregatingindependent chromosomes back in time through a fixed,randomly-generated pedigree from the predictions of the

Preprint submitted to bioRxiv May 22, 2016

not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. The copyright holder for this preprint (which wasthis version posted May 24, 2016. . https://doi.org/10.1101/054957doi: bioRxiv preprint

https://doi.org/10.1101/054957

standard coalescent.That the pedigree should have substantial effects on

coalescence probabilities only during the most recent∼ log2(N) generations is in agreement with other theoret-ical studies of population pedigrees. Chang (1999) foundthat the number of generations until two individuals sharean ancestor in the biparental, pedigree sense converges tolog2(N) as the population size grows. Likewise, Derridaet al. (2000) showed that the distribution of the numberof repetitions in an individual’s pedigree ancestry becomesstationary around log2(N) generations in the past. Thislog2(N)-generation timescale is the natural timescale ofconvergence in pedigrees due to the approximate doublingof the number of possible ancestors each generation backin time.

In each of these studies it is assumed that the populationis panmictic, i.e., that individuals mate with each otheruniformly at random. One phenomenon that may alterthis convergence in pedigrees is population structure withmigration between subpopulations or demes. In a subdi-vided population, the exchange of ancestry between demesdepends on the particular history of migration events em-bedded in the population pedigree. These past migrationevents may be infrequent or irregular enough that the con-vergence in the pedigree depends on the details of the mi-gration history rather than on the reproductive dynamicsunderlying convergence in panmictic populations.

Rohde et al. (2004) studied the sharing of pedigree an-cestry in structured populations and found that populationstructure did not change the log2(N)-scaling of the num-ber of generations until a common ancestor of everyonein the population is reached. Barton and Etheridge(2011) studied the expected number of descendants of anancestral individual, a quantity termed the reproductivevalue, and similarly found that population subdivision didnot much slow the convergence of this quantity over thecourse of generations.

While these results give a general characterization ofhow pedigrees are affected by population structure, a di-rect examination of the coalescent process for loci segre-gating through a fixed pedigree of a structured popula-tion is still needed. Fixing the migration events in thepedigree may produce long-term fluctuations in coalescentprobabilities that make the predictions of the structuredcoalescent break down. The pedigree may also bias in-ference of demographic history. Each sampled locus willhave a relatively great probability of being affected by anymigration events or overlap in ancestry contained in themost recent generations of the sample pedigree, and thusdemographic inference methods that do not take into ac-count the pedigree of the sample may be biased by howthese events shape genetic variation in the sample.

Here, we explore how population structure affects coa-lescence through a fixed population pedigree. Using simu-lations, we investigate how variation in the migration his-tory embedded in the pedigree affects coalescence proba-bilities, and we determine how these pedigree effects de-

pend on population size and migration rate. We also studythe effects of recent admixture on coalescence-time distri-butions and use our findings to develop a simple frameworkfor modeling the sample as a probabilistic mixture of mul-tiple non-admixed ancestries. We demonstrate how thisframework can be incorporated into demographic inferenceby developing a maximum-likelihood method of inferringscaled mutation and migration rates jointly with the re-cent pedigree of the sample. We test this inference ap-proach with simulations, showing that including the pedi-gree in inference corrects a bias that is present when thereis unaccounted-for admixture in the ancestry of the sam-ple.

2. Theory and Results

2.1. Pedigree simulation

Except where otherwise stated, each population wemodel has two demes of constant size, exchanging migrantssymmetrically at a constant rate. This model demon-strates the effects of population structure in one of the thesimplest ways possible and has a relatively simple mathe-matical theory (Wakeley, 2009).

We assume that generations are non-overlapping andthat the population has individuals of two sexes in equalnumber. In each generation, each individual chooses amother uniformly at random from the females of the samedeme with probability 1−m and from the females of theother deme with probability m. Likewise a father is chosenuniformly at random from the males of the same deme withprobability 1 −m and from the males of the other demewith probability m.

All simulations were carried out with coalseam, a pro-gram for simulation of coalescence through randomly-generated population pedigrees. The user provides param-eters such as population size, number of demes, mutationrate, and migration rate, and coalseam simulates a popu-lation pedigree under a Wright-Fisher-like model meetingthe specified conditions. Gene genealogies are constructedby simulating segregation back in time through the pedi-gree, and the resulting genealogies are used to produce sim-ulated genetic loci. Output is in a format similar to thatof the program ms (Hudson, 2002), and various optionsallow the user to simulate and analyze pedigrees featur-ing, for example, recent selective sweeps or fixed amountsof identity by descent and admixture.

The program coalseam is written in C and releasedunder a permissive license. It is available online athttps://github.com/ammodramus/coalseam.

2.2. Structured population pedigrees and probabilities ofcoalescence

In a well-mixed population of size N , the distribution ofcoalescence times for independently segregating loci sam-pled from two individuals shows large fluctuations over thefirst ∼ log2(N) generations depending on the degree of

2


https://doi.org/10.1101/054957

Figure 1: Coalescence time distribution for independently segre-gating loci sampled from two individuals in a panmictic population.The gray line shows the distribution from the pedigree, and the blackline shows the exponential prediction of the standard coalescent. Thepopulation size is N = 500 diploid individuals.

overlap in the pedigree of the two individuals. After thisinitial period, the coalescence probabilities quickly con-verge to the expectation under standard coalescent theory,with small fluctuations around that expectation. (Wake-ley et al., 2012, Fig. 1). The magnitude of these fluctua-tions depends on the population size, but even for small tomoderately sized populations (e.g., N = 500), the expo-nential prediction of the standard coalescent is a good ap-proximation to the true distribution after the first log2(N)generations.

When the population is divided into multiple demes, de-viations from the coalescent probabilities predicted by thestructured coalescent depend on the particular history ofmigration in the population pedigree. The effect of the mi-gration history is especially pronounced when the averagenumber of migration events per generation is of the sameorder as the per-generation pairwise coalescent probability(Fig. 2A). In this migration-limited regime, two lineages indifferent demes have zero probability of coalescing beforea migration event in the pedigree can bring them togetherinto the same deme. This creates large peaks in the coales-cence time distributions for loci segregating independentlythrough the same pedigree, with each peak correspondingto a migration event (Fig. 2A).

Even when the migration rate is greater and there aremany migration events per coalescent event, the pedigreecan still cause coalescence probabilities to differ from thepredictions of the structured coalescent. Under these con-ditions, coalescence is not constrained by individual mi-gration events, but there may be stochastic fluctuationsin the realized migration rate, with some periods experi-encing more migration and others less. These fluctuationscan cause deviations in the predicted coalescence proba-bilities long past the log2(N)-generation timescale foundin well-mixed populations (Figs. 2B, S1, S2). The degreeof these deviations depends on the rate of migration andthe population size, with smaller populations and lowermigration rates causing greater deviations, and deviationsfrom predictions are generally larger for samples taken be-

Figure 2: Distribution of coalescence times for two individuals sam-pled from different demes. In both panels, the black line shows thedistribution calculated from a simulated pedigree, and the purpleline shows the prediction from the structured coalescent. Red verti-cal lines along the horizontal axis show the occurrence of migrationevents in the population, with the relative height representing thetotal reproductive weight (see Barton and Etheridge, 2011) of themigrant individual(s) in that generation. A) Low migration pedi-gree, with M = 4Nm = 0.04 and N = 100. Under these conditions,coalescence is limited by migration events, so there are distinct peaksin the coalescence time distribution corresponding to individual mi-gration events. B) Higher migration pedigree, with M = 0.4 andN = 1000. With the greater migration rate, coalescence is no longerlimited by migration, but stochastic fluctuations in the migrationprocess over time cause deviations away from the standard-coalescentprediction on a timescale longer than log2(N) generations.

tween demes than samples within demes. When there aremany migration events per coalescent event (i.e., whenNm >> 1/N), the predictions of the structured coales-cent fit the observed distributions in pedigrees reasonablywell (Figs. S1–S2).

To investigate the dependence of the coalescence timedistribution on the pedigree more systematically, we sim-ulated 20 replicate population pedigrees for a range of pop-ulations sizes and migration rates. From each pedigree, wesampled two individuals in different demes and calculatedthe distribution of pairwise coalescence times for indepen-dently segregating loci sampled from those two individu-als. We measured the total variation distance from thedistribution predicted under a discrete-time model of co-alescence and migration analogous to the continuous-timestructured coalescent. The total variation distance of twodiscrete distributions P and Q is defined as

DTV (P,Q) =1

2

∑i

|P (i)−Q(i)| . (1)

3


https://doi.org/10.1101/054957

Figure 3: Pairwise coalescence time distributions for samples withadmixed ancestry. Each panel shows the distribution of pairwisecoalescence times for a sample whose pedigree contains some amountof admixture. In each simulation, there are two demes of size N =1000, and the scaled migration rate is 4Nm = 0.1. In each panel,the population pedigree was simulated conditional on the samplehaving the pedigree shown in the panel. The purple line shows thebetween-deme coalescence time distribution that would be expectedin the absence of admixture, and the gold line shows the the mixtureof the within- and between-deme coalescence time distributions thatcorresponds to the degree of admixture. Black lines are numericallycalculated coalescence time distributions for the simulated examplepedigrees.

We found that the total variation distance between the dis-tributions of pairwise coalescence times from pedigrees andthe distributions from standard theory decreases as boththe population size and migration rate increase and that,in general, the total variation distance is more sensitive tothe migration rate than to population size (Fig. S3).

2.3. Admixture and coalescence distributions in pedigrees

As is the case for panmictic populations, the details ofthe recent sample pedigree are most important in deter-mining the patterns of genetic variation in the sample. Inpanmictic populations, overlap in ancestry in the recentpast creates identity-by-descent. In structured popula-tions, an individual may also have recent relatives fromanother deme, resulting in admixed ancestry. When thisoccurs, the distribution of pairwise coalescence times is po-tentially very different from the prediction in the absenceof admixture due to the admixture paths in the pedigreethat lead to a recent change in demes. The degree of thedifference in distributions is directly related to the degreeof admixture, with more recent admixture causing greaterchanges in the coalescence time distribution (Fig. 3).

If an individual in the sample has a migration event inits recent pedigree, there will be some probability that agene copy sampled from this individual was inherited viathe path through the pedigree including this recent migra-tion event. If this is the case, it will appear as if that gene

copy was actually sampled from a deme other than the oneit was sampled from. In this way, the recent sample pedi-gree “reconfigures” the sample by changing the location ofsampled gene copies, with the probabilities of the differ-ent sample reconfigurations depending on the number andtiming of migration events in the recent pedigree.

If many independently segregating loci are sampled fromindividuals having admixed ancestry, some loci will be re-configured by recent migration events, and others will not.In this scenario, the sample itself can be modeled as aprobabilistic mixture of samples taken in different config-urations, and probabilities of coalescence can be calculatedby considering this mixture. For example, consider a sam-ple of independently segregating loci taken from two indi-viduals related by the pedigree shown in Fig. 3C, whereone of two individuals sampled from different demes hasa grandparent from the other deme. The distribution ofpairwise coalescence times for loci sampled from this pairresembles the distribution of Tw/4 + 3Tb/4, where Tb isthe standard between-deme pairwise coalescence time fora structured-coalescent model with two demes, and Tw isthe corresponding within-deme pairwise coalescence time(Fig. 3C). This particular mixture reflects the fact that alineage sampled from the admixed individual follows theadmixture path with probability 1/4.

This sample reconfiguration framework can also be usedto model identity-by-descent (IBD), where overlap amongbranches of the recent sample pedigree causes early coa-lescence with unusually high probability. If the pedigreecauses an IBD event to occur with probability Pr(IBD),then the pairwise coalescence time is a mixture of the stan-dard distribution (without IBD) and instantaneous coa-lescence (on the coalescent timescale) with probabilities1 − Pr(IBD) and Pr(IBD), respectively. If there is bothIBD and admixture in the recent sample pedigree (or ifthere are multiple admixture or IBD events), the samplecan be modeled as a mixture of several sample reconfigu-rations (e.g., Fig. 4).

This approach to modeling the sample implicitly as-sumes that there is some threshold generation separatingthe recent pedigree, which determines the mixture of sam-ple reconfigurations, and the more ancient pedigree, wherethe standard coalescent models are assumed to hold wellenough. The natural boundary between these two periodsis around log2(N) generations, since any pedigree featuremore ancient than that tends to be shared by most or allof the population (making such features “population de-mography”), whereas any features more recent tend to beparticular to the sample.

We note that there is a long history in population ge-netics of modeling genetic variation in pedigrees as a mix-ture of different sample reconfigurations. Wright (1951)wrote the probability of observing a homozygous A1A1

genotype as p2(1 − F ) + pF , where p is the frequency ofA1 in the population and F is essentially the probabilityof IBD calculated from the sample pedigree. This can bethought of as a probability for a mixture of two samples of

4


https://doi.org/10.1101/054957

Figure 4: Distribution of pairwise coalescence times for a samplewhose recent pedigree contains both admixture and IBD. The recentpedigree of the sample is shown, with the two sampled individualslocated at the bottom of the pedigree. The distribution for the sim-ulated pedigree (gray line) is based on numerical coalescence proba-bilities calculated in a pedigree of two demes of size N = 1000 each,with migration rate M = 4Nm = 0.2. The colored lines show mix-tures of the within-deme coalescence time distribution (fTw ) and thebetween-deme distribution (fTb

). The inset shows the probability ofcoalescence during the first five generations; the probability mass atgeneration 1 is predicted by the mixture model accounting for bothIBD and admixture (orange line). The red line shows the distribu-tion if IBD is ignored, and the blue line shows the distribution ifboth IBD and admixture are ignored.

size n = 2 (with probability 1 − F ) and n = 1 (probabil-ity F ). The popular ancestry inference program STRUC-TURE (Pritchard et al., 2000) and related methods sim-ilarly write the likelihood of observed genotypes as a mix-ture over different possible subpopulation origins of thesampled alleles. Here, motivated by our simulations of co-alescence in pedigrees, we explicitly extend this kind ofapproach to modeling coalescence. In the next section, wepropose an approach to inferring population parameterssuch as mutation and migration rates jointly with featuresof the sample pedigree such as recent IBD and admixture.In the Discussion, we further consider the similarities anddifferences between our inference approach and those ofexisting methods.

2.4. Joint inference of the recent sample pedigree and pop-ulation demography

The sample reconfiguration framework for modeling ge-netic variation in pedigrees, described above, can be usedto calculate how admixture or IBD in the recent samplepedigree bias estimators of population-genetic parameters.In Appendix A, we calculate the bias of three estimators ofthe population-scaled mutation rate θ = 4Nµ in a panmic-tic population when the recent sample pedigree containsIBD. In Appendix B, we calculate the bias of a moments-based estimator of M due to recent admixture in the sam-ple pedigree. We use simulations to confirm these calcu-lations (Figs. S5,S6). In both cases, if the recent sample

pedigree is known, it is straightforward to correct theseestimators to eliminate bias.

It is uncommon that the recent pedigree of the sampleis known, however, and if it is assumed known, it is oftenestimated from the same data that is used to infer de-mographic parameters. Ideally, one would infer long-termdemographic history jointly with sample-specific featuresof the pedigree. In this section, we develop a maximum-likelihood approach to inferring IBD and admixture jointlywith scaled mutation and migration rates. The methoduses the approach proposed in the previous section: thesample pedigree defines some set of possible outcomes ofMendelian segregation in recent generations, and the re-sulting, reconfigured sample is modeled by the standardcoalescent process.

We model a population with two demes each of size N ,with each individual having probability m of migrating tothe other deme in each generation. We rescale time by Nso that the rate of coalescence within a deme is 1 and therescaled rate of migration per lineage is M/2 = 2Nm. Weassume that we have sampled two copies of each locus fromeach of n1 (diploid) individuals from deme 1 and n2 indi-viduals from deme 2. We write the total diploid samplesize as n1 + n2 = n so that the total number of sequencessampled at each locus is 2n. We index our sequences withIn := 1m, 1p, 2m, 2p, . . . , nm, np, where im and ip indexthe maternal and paternal sequences sampled from indi-vidual i. These are simply notational conventions; we donot assume that these maternal and paternal designationsare known or observed.

Each recent pedigree P has some set of possible out-comes of segregation through the recent past, involvingcoalescence of lineages (IBD) and movement of lineagesbetween demes (admixture). The set of these sample re-configurations is denoted R(P), and each reconfigurationr ∈ R(P) is a partition of In, with the groups in r repre-senting the lineages that survive after segregation back intime through the recent pedigree. Each group in a recon-figuration is also labeled with the deme in which the corre-sponding lineage is found segregation back in time throughthe recent pedigree. Corresponding to each sample recon-figuration r ∈ R(P), there is a probability Pr(r | P) ofthat sample reconfiguration being the outcome of segrega-tion back in time through the recent pedigree.

The data X = X1,X2, . . . ,XL consist of sequencedata at L loci. The data at locus i are represented by the

sequences Xi = X(a)i,1 , X

(b)i,1 , X

(a)i,2 , X

(b)i,2 , . . . , X

(a)i,n , X

(b)i,n ,

where X(a)i,j and X

(b)i,j are the two sequences at locus i from

individual j. We label them (a) and (b) because we as-sume that they are of unknown parental origin. In orderto make calculation of sampling probabilities feasible, weassume that each sequence evolves under the infinite-sitesmutation model and can thus be represented by a binarysequence. We further assume that there is free recombina-tion between loci and no recombination within loci.

Our goal is to find the θ, M , and P that maximize the

5


https://doi.org/10.1101/054957

likelihood

L(P, θ,M |X) = Pr(X | P; θ,M) =

L∏i=1

∑r∈R(P)

Pr(Xi | r; θ,M) Pr(r | P).(2)

Probabilities are multiplied across independently segregat-ing loci because their ancestries are independent condi-tional on the pedigree.

In order to calculate Pr(Xi | r; θ,M), it is necessary toconsider all the ways the sequences Xi could have beeninherited maternally versus paternally, since we assumethat we do not know which sequence is maternal and whichis paternal. For sequences Xi, let Λ(Xi) be the set of allpossible ways of labeling Xi as maternal and paternal,thus associating each sequence with an index in In. Thesampling probability of the data at locus i is then

Pr(Xi | r; θ,M) =1

2n

∑λ∈Λ(Xi)

Pr(Xi | λ, r; θ,M), (3)

since there are 2n ways that the Xi could have segregatedas maternal and paternal alleles, and each is equally likelyto have occurred.

Together the reconfiguration r ∈ R(P) and thematernal-paternal labeling λ ∈ Λ(Xi) imply a partitionP(Xi, r, λ) of the sequences Xi corresponding to the par-tition of sequence indices represented by r. For each grouph ∈ P(Xi, r, λ), there is a group g ∈ r that can be mappedonto h such that 1) each index i ∈ g indexes a distinctsequence in h sampled from the individual indexed by i,and 2) the deme labeling of g matches the deme label-ing of h. Denote the unique elements of a set A as A6=.The conditional sampling probability of the sequences Xi

given maternal-paternal labeling λ ∈ Λ(Xi) and samplereconfiguration r ∈ R(P) is

Pr(Xi | λ, r; θ,M) = Pr(P(Xi, r, λ); θ,M) =φ(h 6= : h ∈ P(Xi, r, λ); θ,M) if |h 6=| = 1

∀h ∈ P(Xi, r, λ)

0 otherwise,

(4)

where each h ∈ P(Xi, r, λ) is one of the non-empty subsetsin the partitioned sequences, |h 6=| is the number of uniqueelements in such a subset, h6= : h ∈ P(Xi, r, λ) is the“reduced” set of sequences, such that each subset in thepartition is replaced the unique elements in the subset, andφ(h 6= : g ∈ P(Xi, r, λ); θ,M) is the standard infinite-sites sampling probability of the sample after it has beenreconfigured by the recent pedigree. This sampling proba-bility can be calculated numerically using a dynamic pro-gramming approach (Griffiths and Tavare, 1994; Wu,2010, see below).

In other words, conditional on certain sequences beingIBD (i.e., they are in the same group in the partitioned se-quences), the sampling probability is the standard infinite-sites probability of the set of sequences with duplicate IBDsequences removed and the deme labelings of the differentgroups made to match any admixture events that may haveoccurred. If any of the sequences designated as IBD arenot in fact identical in sequence, the sampling probabilityfor that reconfiguration and maternal-paternal labeling iszero. This is equivalent to assuming that no mutation oc-curs in the recent part of the pedigree.

Together, (2), (3), and (4) give the overall joint log-likelihood of mutation rate θ, migration rate M , and pedi-gree P given sequences X:

LL(θ,M,P |X) =

L∑i=1

log

∑r∈R(P)

Pr(r | P)∑

λ∈Λ(Xi,r)

Pr(P(Xi, r, λ); θ,M)

−nL log(2)

(5)

Our goal is to maximize (5) over θ, M , and P in orderto estimate these parameters. One naive approach wouldbe to generate all possible recent sample pedigrees andmaximize the log-likelihood conditional on each pedigreein turn. However, the number of pedigrees to consider isprohibitively large even if only the first few generationsback in time are considered. Many sample pedigrees willcontain many IBD or admixture events and thus be un-likely to occur in nature, and in many populations, it ismore probable that the sample will have few IBD or admix-ture events in the very recent pedigree, if any. With thisin mind, we consider only pedigrees containing no morethan two events, whether they be IBD events or admixtureevents. We further limit the number of pedigrees by con-sidering only the past three generations. Besides reducingthe number of pedigrees requiring calculations, this alsolimits consideration to pedigrees having the greatest effecton genetic variation. Finally, since we assume that we donot know the parental origin of each sequence, we furtherreduce the number of pedigrees to consider by evaluatingonly pedigrees that are unique up to labeling of ancestorsas maternal and paternal.

In a two-deme population, each recent sample pedigreewith two or fewer IBD or admixture events has the shapeof one of the pedigrees shown in Figure S4. There areat most 21 distinct shapes of pedigrees with two or fewerevents, and for each such pedigree shape, there exist somenumber of pedigrees with unique labelings of the sam-pled individuals and timings of the events in the pedi-gree. (Fewer distinct shapes are possible if the samplesize is too small to permit certain shapes.) Table 1 givesthe number of distinct pedigrees that must be consideredfor different sample sizes and numbers of past generationsconsidered. We note that we consider only pedigrees with

6


https://doi.org/10.1101/054957

Table 1: Number of distinct pedigrees with two or fewer IBD oradmixture events. Pedigrees that differ only in maternal-paternallabeling of individuals are not counted as distinct.

generations sample sizen = 1 n = 2 n = 3 n = 4

g = 2 16 123 434 1109g = 3 41 328 1144 2879g = 4 78 631 2190 5477

non-overlapping generations and the sampled individualsfound in the current generation. This is sufficient for theWright-Fisher-like model of pedigrees that we model here,but real pedigrees will not in general satisfy these criteria.Allowing overlapping generations will require considera-tion of many additional pedigrees and introduce problemsof identifiability, with multiple pedigrees having the sameset of reconfigurations with the same probabilities. We donot explore these issues here.

To calculate the standard sampling probabilities neededin (4), we use the dynamic-programming method describedby Wu (2010). In principle, it should be possible to cal-culate the log-likelihood of all pedigrees simultaneously,since any reconfiguration of the sample by recent IBD oradmixture must correspond to one of the ancestral config-urations in the recursion solved by Wu’s (2010) method(see also Griffiths and Tavare, 1994). Thus, for par-ticular values of θ and M , after solving the ancestral recur-sion only once (and storing the sampling probabilities ofall ancestral configurations), the likelihood of any pedigreecan be found by extracting the relevant probabilities fromthe recursion. However, in order to take this approach tomaximize the log-likelihood, it is necessary to solve therecursion on a large grid of θ and M . In practice, wefind that it is faster to maximize the log-likelihood sep-arately for each pedigree, using standard derivative-freenumerical optimization procedures to find the M and θthat maximize the log-likelihood for the pedigree. We useordered sampling probabilities throughout, since the typi-cal unordered probabilities would be inappropriate in thiscontext due to the partial ordering of the sample by thepedigree.

To test our inference method we simulated datasets of1000 independently segregating loci, generated by simu-lating coalescence through a randomly generated pedigreeof a two-deme population with deme size N = 1000 andmigration rate M ∈ 0.2, 2.0. We sampled one individ-ual (two sequences) from each deme. Sequence data weregenerated by placing mutations on the simulated gene ge-nealogies according to the infinite-sites mutation modelwith rate θ/2 = 0.5 (when M = 0.2) or θ/2 = 1.0 (whenM = 2.0). In order to investigate the effects of admix-ture on the estimation on θ and M , each replicate datasetwas conditioned upon having one of three different samplepedigrees with differing amounts of admixture (see Fig. 5).We calculated maximum-likelihood estimates of θ and M

for each of the 41 distinct pedigrees containing two or fewerIBD or admixture events occurring in the past three gener-ations. We compared these estimates to the estimates thatwould be obtained from a similar maximum-likelihood pro-cedure that ignores the pedigree (i.e., assuming the nullpedigree of no sample reconfiguration).

When there was recent admixture in the sample, assum-ing the null pedigree to be the true pedigree produced abias towards overestimation of the migration rate (Fig. 5),since the early probability of migration via the admixturepath must be accommodated by an increase in the migra-tion rate. For this reason, the overestimation of the mi-gration rate was greater when the degree of admixture wasgreater. The mutation rate was also overestimated whenthe admixture in the pedigree was ignored, presumably be-cause migration via the admixture path did not decreaseallelic diversity as much as the overestimated migrationrate should. Including the pedigree as a free parameterin the estimation corrected this biased estimation of M inthe presence of admixed ancestry in the sample. Estimatesfrom simulations of samples lacking any features in the re-cent pedigree produced approximately unbiased estimatesof θ and M (Fig. 5).

The pedigree was not inferred as reliably as the muta-tion and migration rates (Fig. 6). When the simulatedpedigree contained admixture, the estimated pedigree wasthe correct pedigree (out of 41 possible pedigrees) roughlyhalf of the time. For pedigrees with no admixture and noIBD, the correct pedigree was inferred about one third ofthe time. In addition to calculating a maximum-likelihoodpedigree, it is possible to construct an approximate 95%confidence set of pedigrees using the fact that the maxi-mum of the log-likelihood is approximately χ2 distributedwhen the number of loci is large. These pedigree confi-dence sets contained the true pedigree ∼ 88− 99% of thetime, depending on the true sample pedigree, mutationrate, and migration rate. A log-likelihood ratio test hasnearly perfect power to reject the null pedigree for thesimulations with the lesser migration rate; for simulationswith the greater migration rate the power depended onthe degree of admixture, with more recent admixture pro-ducing greater power to reject the null pedigree (Fig. 6).Type I error rates for simulations where the null pedigreeis the true pedigree were close to α = 0.05.

We also simulated datasets of 1000 loci sampled fromindividuals with completely random pedigrees (i.e., sam-pled without conditioning on any admixture in the recentsample pedigree) in a small two-deme population of sizeN = 50 per deme with one of two different migrationrates (M ∈ 0.2, 2.0). Whether or not the pedigree wasincluded as a free parameter mostly had little effect onthe estimates of θ and M (Fig. 7). However, in the fewcases when the sample pedigree included recent admixture,the estimates were biased when the sampled pedigree wasnot considered as a free parameter. Inferring the pedigreetogether with the other parameters corrected this bias.(Fig. 7).

7


https://doi.org/10.1101/054957

with pedigree

without pedigree

true values

Figure 5: Maximum-likelihood mutation rates and migration rates for datasets simulated through different sample pedigrees. Each pointdepicts the maximum-likelihood estimates of θ and M for a particular simulation. Orange points show estimates obtained when the pedigreeis included as a free parameter, and purple points show estimates obtained when the pedigree is assumed to have no effect on the data. Inthe first two columns, one of the sampled individuals is conditioned upon having a relative from the other deme, and in the third columnthe data are generated from completely random pedigrees. In each panel the true parameter values are shown with a solid white circle, andhorizontal and vertical lines show means across replicates. Gray lines connect estimates calculated from the same dataset.

3. Discussion

Here we have explored the effects of fixed migrationevents in the population pedigree on the patterns of co-alescence of independently segregating loci. In contrastto the case of panmictic populations, in structured popu-lations the population pedigree can influence coalescencewell beyond the time scale of ∼ log2(N) generations in thepast. These effects are greater when the migration rateis small and are particularly pronounced when the totalnumber of migration events occurring per generation is ofthe same order as the coalescence probability. When mi-gration occurs more frequently than this (and there is noadmixture in the recent sample pedigree), the particularhistory of migration events embedded in the populationpedigree has less of an effect on coalescence, and the co-alescence distributions based on the structured coalescentserve as good approximations to coalescent distributionswithin pedigrees.

We have also proposed a framework for inferring demo-graphic parameters jointly with the recent pedigree of thesample. This framework considers how recent migrationand shared ancestry events reconfigure the sample by mov-ing lineages between demes and coalescing lineages. Theinferred sample pedigree is the sample pedigree that hasthe maximum-likelihood mixture distribution of sample re-configurations. In our implementation of this framework,we consider only sample pedigrees having two or fewerevents, which must have occurred in the most recent threegenerations. At the expense of computational runtimes,the set of possible sample pedigrees could be expandedto include pedigrees with more events extending a greater

number of generations into the past, but as these limitsare extended, the number of pedigrees to consider growsrapidly. Another approach one could take would be to con-sider all of the reconfigurations (rather than sample pedi-grees) with fewer than some maximum number of differ-ences from the original sample configuration, and find themixture of reconfigurations that maximizes the likelihoodof the data, without any reference to the pedigree thatcreates the mixture. Such an approach may allow greaterflexibility in modeling the effects of the recent pedigree,but searching the space of possible mixtures of reconfig-urations — the k-simplex, if k is the number of possiblereconfigurations — would likely be more challenging thanmaximizing over the finite set of sample pedigrees in themethod we have implemented.

The sample reconfiguration inference framework is com-plementary to existing procedures for inferring recent ad-mixture and relatedness in structured populations. Thepopular program STRUCTURE (Pritchard et al., 2000)and related methods (Raj et al., 2014; Alexander et al.,2009; Tang et al., 2005) are powerful and flexible toolsfor inferring admixture and population structure. Like-wise, the inference tools RelateAdmix (Moltke and Al-brechtsen, 2014), REAP (Thornton et al., 2012), andKING-robust (Manichaikul et al., 2010) all offer solu-tions to the problem of inferring relatedness in the pres-ence of population structure and admixture. Perhaps themost similar in scope is the method of Wilson and Ran-nala (2003), which uses inferred ancestry proportions toestimate migration rates in the most recent generations.For input, each of these methods take genotypes at poly-

8


https://doi.org/10.1101/054957

Figure 6: Inference of sample pedigrees. For simulations of 1000infinite-sites loci, with θ = 0.5 and M = 0.2 (A) or θ = 1.0 andM = 2.0 (B), different measurements of the accuracy and power ofpedigree inference are shown. The conditioned-upon sample pedi-grees are shown at the bottom of the figure. Blue bars show theproportion of simulations in which the maximum-likelihood pedigreewas the true pedigree. Purple bars show the proportion of simula-tions where it was inferred that sampled individuals had admixedancestry. Green bars show the proportion of simulations in whichthe true pedigree was found within the approximate 95% confidenceset of pedigrees, and pink bars show the proportion of simulations inwhich the null pedigree is rejected by a log-likelihood ratio test.

Figure 7: Maximum-likelihood mutation rates and migration ratesfor datasets of 1000 loci segregated through random pedigrees simu-lated in a two-deme population with deme size N = 50. Each pointdepicts the maximum-likelihood estimates of θ and M for a partic-ular simulation. Orange points show estimates obtained when thepedigree is included as a free parameter, and purple points show es-timates obtained when the pedigree is assumed to have no effect onthe data. True parameter values are shown with white circles, andhorizontal and vertical lines show means across replicates. Gray linesconnect estimates calculated from the same dataset.

9


https://doi.org/10.1101/054957

morphic sites, often biallelic SNPs, that are assumed tosegregate independently. Likelihoods are calculated fromthe probabilities of observing the observed genotypes un-der the rules of Hardy-Weinberg equilibrium. These meth-ods are well suited for samples of a large number of SNPloci sampled from a large number of individuals. The infer-ence procedure we have implemented, on the other hand,is capable of handling a sample of only a few individu-als (n ≈ 4), and likelihood calculations in our methodare based on the coalescent in an explicit population ge-netic model, the parameters of which being the primaryobjects of inference. The pedigree is also explicitly mod-eled. While we have shown that our approach works wellwhen its assumptions hold, if the narrow assumptions ofour model do not hold, or if the primary goal of inferenceis to infer recent features of the sample pedigree per se,other, more flexible methods are likely to perform better.

Underlying our inference method is a hybrid approachto modeling the coalescent. Probabilities of coalescenceare determined by the sample pedigree in the recent past,and then the standard coalescent is used to model the moredistant past. This is similar to how Bhaskar et al. (2014)modeled coalescence when the sample size approaches thepopulation size. In such a scenario, they suggest usinga discrete-time Wright-Fisher model to model coalescencefor the first few generations back in time and then use thestandard coalescent model after the number of survivingancestral lineages becomes much less than the populationsize. We note that in a situation where the sample sizenears the population size in a diploid population, therewill be numerous common ancestor events and admixtureevents in the recent sample pedigree, so it may be im-portant to consider the pedigree when genetic variation issampled from a fixed set of individuals at independentlysegregating loci.

The sample reconfiguration framework could be ex-tended to models that allow the demography of the popu-lation to vary over time (e.g., Gutenkunst et al., 2009;Kamm et al., 2015). In such an application, if only a fewindividuals are sampled, it would be important to distin-guish between the effects of very recent events that arelikely particular to the sample and the effects of eventsthat are shared by all individuals in the population. Thelatter category of events are more naturally considered de-mographic history. On the other hand, if a sizable fractionof the population is sampled, inferred pedigree featuresmay be used to learn more directly about the demographyof the population in the last few generations. As samplesizes increase from the tens of thousands into the hun-dreds of thousands and millions (Stephens et al., 2015),it will become more and more possible to reconstruct large(but sparse) pedigrees that are directly informative aboutrecent demographic processes.

Unexpected close relatedness is frequently found in largegenomic datasets (e.g. Gazal et al., 2015; Pembertonet al., 2010; Rosenberg, 2006). It is common practice toremove closely related individuals (and in some cases, in-

dividuals with admixed ancestry) from the sample priorto analysis, but this unnecessarily reduces the amountof information that is available for inference. What isneeded is a fully integrative method of making inferencesfrom pedigrees and genetic variation, properly incorporat-ing information about both the recent past contained inthe sample pedigree and the more distant past that is themore typical domain of population genetic demographicinference. Here, by performing simulations of coalescencethrough pedigrees, we have justified a sample reconfigu-ration framework for modeling coalescence in pedigrees,and we have demonstrated how this can be incorporatedinto coalescent-based demographic inference in order toproduce unbiased estimates of demographic parameterseven when there is recent relatedness or admixed ances-try amongst the sampled individuals.

4. Acknowledgments

We thank Seungsoo Kim, Mark Martinez, Janet Song,and Katherine Xue for their their assistance during theearly phases of this project. We also thank Leandra Kingand Julia Palacios for helpful discussion and the ResearchComputing Group at Harvard University for computingresources.

5. References

Alexander, D. H., J. Novembre, and K. Lange, 2009 Fast model-based estimation of ancestry in unrelated individuals. GenomeResearch 19: 1655–1664.

Barton, N. H., and A. M. Etheridge, 2011 The relation betweenreproductive value and genetic contribution. Genetics 188: 953–973.

Bhaskar, A., A. G. Clark, and Y. S. Song, 2014 Distortion ofgenealogical properties when the sample is very large. Proceedingsof the National Academy of Sciences 111: 2385–2390.

Chang, J. T., 1999 Recent common ancestors of all present-dayindividuals. Advances in Applied Probability 31: 1002–1026.

Derrida, B., S. C. Manrubia, and D. H. Zenette, 2000 On thegenealogy of a population of biparental individuals. Journal ofTheoretical Biology 203: 303–315.

Gazal, S., M. Sahbatou, M.-C. Babron, E. Genin, and A.-L.Leutenegger, 2015 High level of inbreeding in final phase of 1000Genomes Project. Scientific Reports 5.

Griffiths, R. C., and S. Tavare, 1994 Ancestral Inference in Pop-ulation Genetics. Statistical Science 9: 307–319.

Gutenkunst, R. N., R. D. Hernandez, S. H. Williamson, andC. D. Bustamante, 2009 Inferring the Joint Demographic Historyof Multiple Populations from Multidimensional SNP FrequencyData. PLoS Genetics 5: e1000695.

Hein, J., M. H. Schierup, and C. Wiuf, 2005 Gene Genealogies,Variation and Evolution: A Primer in Coalescent Theory. OxfordUniversity Press, Oxford, 1 edition.

Hudson, R. R., 1983 Testing the constant-rate neutral allele modelwith protein sequence data. Evolution 37: 203–217.

Hudson, R. R., 2002 Generating samples under a Wright–Fisherneutral model of genetic variation. Bioinformatics 18: 337–338.

Kamm, J. A., J. Terhorst, and Y. S. Song, 2015 Efficient computa-tion of the joint sample frequency spectra for multiple populations.arXiv:1503.01133 .

Kingman, J. F. C., 1982a The coalescent. Stochastic Processes andtheir Applications 13: 235–248.

10


https://doi.org/10.1101/054957

Kingman, J. F. C., 1982b On the genealogy of large populations.Journal of Applied Probability 19: 27–43.

Li, H., and R. Durbin, 2011 Inference of human population historyfrom individual whole-genome sequences. Nature 475: 493–496.

Manichaikul, A., J. C. Mychaleckyj, S. S. Rich, K. Daly,M. Sale, et al., 2010 Robust relationship inference in genome-wide association studies. Bioinformatics 26: 2867–2873.

Moltke, I., and A. Albrechtsen, 2014 RelateAdmix: a soft-ware tool for estimating relatedness between admixed individuals.Bioinformatics 30: 1027–1028.

Pemberton, T. J., C. Wang, J. Z. Li, and N. A. Rosenberg, 2010Inference of unexpected genetic relatedness among individuals inHapMap Phase III. The American Journal of Human Genetics87: 457–464.

Pritchard, J. K., M. Stephens, and P. Donnelly, 2000 Inferenceof population structure using multilocus genotype data. Genetics155: 945–959.

Raj, A., M. Stephens, and J. K. Pritchard, 2014 fastSTRUC-TURE: Variational inference of population structure in large SNPdata sets. Genetics 197: 573–589.

Rohde, D. L. T., S. Olson, and J. T. Chang, 2004 Modelling therecent common ancestry of all living humans. Nature 431: 562–566.

Rosenberg, N. A., 2006 Standardized subsets of the HGDP-CEPHHuman Genome Diversity Cell Line Panel, accounting for atypicaland duplicated samples and pairs of close relatives. Annals ofHuman Genetics 70: 841–847.

Schiffels, S., and R. Durbin, 2014 Inferring human population sizeand separation history from multiple genome sequences. NatureGenetics 46: 919–925.

Sheehan, S., K. Harris, and Y. S. Song, 2013 Estimating vari-able effective population sizes from multiple genomes: a sequen-tially Markov conditional sampling distribution approach. Genet-ics 194: 647–662.

Stephens, Z. D., S. Y. Lee, F. Faghri, R. H. Campbell, C. Zhai,et al., 2015 Big Data: Astronomical or Genomical? PLOS Biol13: e1002195.

Tajima, F., 1983 Evolutionary relationship of DNA sequences infinite populations. Genetics 105: 437–460.

Tang, H., J. Peng, P. Wang, and N. J. Risch, 2005 Estimation ofindividual admixture: Analytical and study design considerations.Genetic Epidemiology 28: 289–301.

Thornton, T., H. Tang, T. Hoffmann, H. Ochs-Balcom,B. Caan, et al., 2012 Estimating kinship in admixed populations.The American Journal of Human Genetics 91: 122–138.

Wakeley, J., 2009 Coalescent Theory: An Introduction. Robertsand Co., Greenwood Village, CO.

Wakeley, J., L. King, B. S. Low, and S. Ramachandran, 2012Gene Genealogies Within a Fixed Pedigree, and the Robustnessof Kingmans Coalescent. Genetics 190: 1433–1445.

Watterson, G., 1975 On the number of segregating sites in geneticalmodels without recombination. Theoretical Population Biology 7:256–276.

Wilson, G. A., and B. Rannala, 2003 Bayesian Inference of Re-cent Migration Rates Using Multilocus Genotypes. Genetics 163:1177–1191.

Wright, S., 1951 The genetical structure of populations. Annals ofEugenics 15: 323–354.

Wu, Y., 2010 Exact computation of coalescent likelihood for pan-mictic and subdivided populations under the infinite sites model.IEEE Transactions on Computational Biology and Bioinformatics7: 611–618.

6. Appendix A: Pedigrees and biased estimators ofθ

Estimates of the population-scaled mutation rate θ =4Nµ will be downwardly biased if there is recent IBD in thesample, since sequences will be identical with an artificially

inflated probability, and this resembles coalescence prior toany mutation between the two identical sequences.

Suppose that we sample two copies of a DNA sequencefrom each of n diploid individuals from a panmictic pop-ulation. As in the main text, we index these sequenceswith In := 1m, 1p, 2m, 2p, . . . , nm, np. Let PIn be theset of partitions of In. Each pedigree P induces a setof sample reconfigurations R(P) ⊆ PIn , where each par-tition r ∈ R(P) represents a possible outcome of seg-regation through the recent sample pedigree. Each re-configuration r ∈ R(P) contains |r| non-empty, disjointsubsets, each representing a distinct lineage that survivesafter segregation through the recent pedigree. Associ-ated with each pedigree is also a probability distributionPr(r | P), r ∈ R(P) representing the Mendelian probabil-ities of the different sample reconfigurations.

We consider the bias of three estimators of θ that areunbiased in the absence of recent IBD. One estimator of θwe consider is Watterson’s (1975) estimator

θS =

∑Li=1 S

(i)

anL, (6)

where n is the (haploid) sample size, S(i) is the number ofsegregating sites at locus i, L is the number of loci, andan =

∑n−1i=1 1/i. The expected value of θS given pedigree

P is

E[θS | P

]=

∑r∈R(P)

E[θS | r

]Pr(r | P)

=∑

r∈R(P)

E

[S(1)

an| r]

Pr(r | P)

= θ∑

r∈R(P)

a|r|

anPr(r | R(P)).

(7)

This follows from the fact that when there are |r| lineagessurviving the recent pedigree, the expected number of seg-

regating sites for that sample is θ∑|r|−1i=1

θi = θa|r|.

A second estimator of θ is π, the mean number of dif-ferences between all pairs of sequences in a sample, whichcan be written in terms of the site-frequency spectrum:

π =1(n2

) n−1∑i=1

i(n− i)ξi, (8)

where ξi is the number of segregating sites present in isequences in the sample.

To calculate the expected value of π given P, it is nec-essary to consider how IBD in the recent pedigree changesthe site frequency spectrum. Define S(n) := A ⊆ Ω : Ω ∈PIn and ψ : PIn × N→ S(n) as

ψ(Ω, i) := ω ⊆ Ω :∑g∈ω|g| = i. (9)

11


https://doi.org/10.1101/054957

That is, ψ(Ω, i) is the set of all subsets of the partition Ωsuch that the total size of all the groups in each subset isi. For example, for n = 3 and

Ω = 1m, 1p, 2m , 2p, 3m , 3p ,

ψ(Ω, 1) = 3pψ(Ω, 2) = 2p, 3mψ(Ω, 3) = 1m, 1p, 2m, 2p, 3m, 3pψ(Ω, 4) = 1m, 1p, 2m, 3pψ(Ω, 5) = 1m, 1p, 2m, 2p, 3m.

Then the expectation of ξi given reconfiguration r ∈R(P) is

E[ξi | r

]=

∑ω∈ψ(r,i)

θ

|ω|(|r||ω|) , (10)

since each segregating site that is present in i present-daylineages must have occurred on the branch ancestral tothe post-IBD lineages represented by some ω ∈ ψ(r, i).The expected number of mutations occurring on a branchsubtending |ω| lineages is θ/|ω|, and the expected fractionof such mutations that occur on the branch ancestral tothe lineages in ω is 1/

(|r||ω|), by exchangeability. This gives

the expectation of π conditional on the pedigree:

E [π | P] =∑

r∈R(P)

E [π | r] Pr(r | P)

=1(n2

) ∑R∈P

Pr(R | P)

n−1∑i=1

i(n− i) E[ξi | r]

=θ(n2

) ∑r∈R(P)

Pr(r | P)n−1∑i=1

i(n− i)∑

ω∈ψ(r,i)

1

|ω|(|R||w|)

(11)

A third estimator of θ is ξ1, the number of singletonsin the sample. The conditional expectation of ξ1 given apedigree P is

E[ξ1 | P

]= θ

∑r∈R(P)

|ψ(r, 1)||R(P)|

Pr(r | P), (12)

since only those mutations that occur on lineages that havenot coalesced with any other lineages in the early pedigreecan produce singletons.

To validate these calculations, we performed simulationsof 200 loci sampled from individuals whose pedigree in-cludes some degree of IBD or inbreeding. The simulationsconfirm the calculated biases for the different estimatorsof θ (Fig. S5).

7. Appendix B: Pedigrees and biased estimators ofM

In a constant-sized structured population with twodemes and a constant rate of migration M = 4Nm be-tween demes, the expected within-deme and between-demepairwise coalescence times are

E[Tw] = 2 (13)

E[Tb] = 2 + 1/M, (14)

Let πw and πb be the within-deme and between-deme meanpairwise diversity, respectively. Since E[πw] = θE[Tw] andE[πb] = θE[Tb], one estimator of M is

M =πw

2(πb − πw). (15)

If some individuals in the sample are recently admixed,M will be biased. In general it is not possible to calculateE[M ], but it can be approximated by

E[M ] ≈ E[πw]

2(E[πb]− E[πw]). (16)

We sample two sequences from each of n1 individualsfrom deme 1 and n2 individuals from deme 2, definingn = n1 + n2 as the total (diploid) sample size. The sam-ple is again indexed by In = 1m, 1p, . . . , nm, np, and weassume that the first 2n1 of these indices correspond to se-quences sampled from deme 1 and the last 2n2 from deme2.

In the context of a two-deme population, each groupin the partitioned sample r ∈ R(P) is labeled 1 or 2 toindicate which deme the lineage is found in after segrega-tion back in time through the recent sample pedigree. Fortwo-deme reconfiguration r, let d(r, i, j), i, j ∈ 1, 2, be afunction that gives the number of lineages originally sam-pled from deme i that are found in deme j after segregationthrough the recent sample pedigree.

Assume that the recent sample pedigree contains ad-mixture but no IBD. In this case, we can write the ex-pectations of πw and πb conditional on reconfiguration ras

E [πw | r] =1(

2n1

2

)+(

2n2

2

)×θE[Tw]

2∑i=1

(d(r, i, i)

2

)+

θE[Tb] (d(r, 1, 1)d(r, 1, 2) + d(r, 2, 2)d(r, 2, 1))

(17)

12


https://doi.org/10.1101/054957

and

E [πb | r] =1

4n1n2×

θE[Tb](d(r, 1, 1)d(r, 2, 2) + d(r, 1, 2)d(r, 2, 1)

)+

θE[Tw] (d(r, 1, 2)d(r, 2, 2) + d(r, 2, 1)d(r, 1, 1))

.

(18)

The approximate expectation of M conditional on apedigree P can be calculated using (17) and (18) togetherwith

E[πw | P] =∑

r∈R(P)

E[πw | r] Pr(r | P)

andE[πb | P] =

∑r∈R(P)

E[πb | r] Pr(r | P).

Simulations of infinite-sites loci taken from samples witha single admixed ancestor confirm these calculations(Fig. S6). This method of approximating E[M | P] couldbe extended to accommodate recent sample pedigrees thatcontain both IBD and admixture, but we do not pursuethis here.

13


https://doi.org/10.1101/054957

8. Supplementary Material

Figure S1: Distributions of coalescence times in two-deme populations with N = 100 individuals in each deme. Each panel shows adistribution of coalescence times for a particular value of the migration rate M = 4Nm. For panels in the left column, two individuals weresampled from the same deme (“within-deme” sampling), and in the right column two individuals were sampled from different demes.

14


https://doi.org/10.1101/054957

Figure S2: Distributions of coalescence times in two-deme populations with N = 1000 individuals in each deme. Each panel shows adistribution of coalescence times for a particular value of the migration rate M = 4Nm. For panels in the left column, two individuals weresampled from the same deme (“within-deme” sampling), and in the right column two individuals were sampled from different demes.

15


https://doi.org/10.1101/054957

Figure S3: Total variation distance between pairwise coalescence time distributions in pedigrees versus standard theory. For each point onthe grid, the total variation distance from the prediction of standard theory was averaged over 20 pedigrees with the corresponding N andM .

Figure S4: Distinct Wright-Fisher pedigree shapes with two or fewer IBD or admixture events in a two-deme population. Only relationshipsinvolved in IBD or admixture events are shown. All pedigrees have non-overlapping generations, and sampled individuals (white circles) areliving in the present generation. To produce all distinct pedigrees, it is necessary to consider all the ways of indexing the individuals involvedin the IBD and admixture events, as well as the all of the possible timings of these events.

16


https://doi.org/10.1101/054957

0.6

0.8

1.0

1.2

−−− −−− −−− −−−− − − − − −

− − − − − −

−prediction− sample mean +/- 2 s.e.m

Figure S5: Estimates of θ = 4Nµ based on 100 replicate simulations of 200 loci segregating through sample pedigrees featuring identity bydescent. Blue lines indicate theoretical predictions for individual pedigrees and estimators. Red horizontal lines indicate sample means, andred vertical lines indicate twice the standard error of the mean. The true value of θ = 1.0 is indicated by the dashed line. Note that for thefirst pedigree, with n = 2, the three estimators are equivalent.

0.0

0.5

1.0

1.5

−− −− −− −−...

− prediction

− sample mean +/- 2 s.e.m

Figure S6: Estimates of M from the estimator given by (15) in replicate simulated genetic datasets of 100 loci segregating through samplepedigrees containing recent admixture. Blue lines indicate theoretical predictions for individual pedigrees. Red horizontal lines indicatesample means, and red vertical lines indicate twice the standard error of the mean. The true value of M = 0.2 is indicated by the dashedline. P (A) indicates the probability of admixture in the indicated pedigrees.

17


https://doi.org/10.1101/054957

population structure and coalescence in pedigrees ... · to construct gene genealogies, the...

Documents