However, in large populations, most new branches won’t spread throughout the population and many die off; i.e. some Y-chromosomes don’t get passed one when Men don’t father sons.
The probability of two y-chromosomes to coalesce in the previous generation = 1 / (Ne/2). Ne = effective population size, and it’s divided by 2 because ~50% of people have a y-chromosome. But we can simplify the formula by defining Ny as the Y-chromosomal population size, in which case P = 1/Ny. Conversely, probability of NOT coalescence in previous generation = 1-1/Ny. The probability of NOT coalescence within the previous t number of generations = 1-1/Ny^t. Hence, the probability of coalescence within the previous t number of generations = 1-(1-1/Ny^t)
On a graph, it looks like this:
Basically, this illustrates that with larger population sizes, the more generations it takes for two lineages to coalesce. But we don’t have just two y-chromosomes in a population. What about all the others? When there are multiple individuals, there are many more ways for any two of them to coalesce. In this case, you need to add the following formula to the equation: k(k-1)/2. Meaning, if you only have two y-chromosome branches to consider, it means we only have 2(2-1)/2=1 unique pair that can coalesce. However, if we have three branches, then that number is 3(3-1)/2=3. Thus, the probability that none of the k number of y-chromosomes coalesce one generation ago = 1-(1-1/Ny^(k(k-1)/2)). And adding the t (umber of generations) we get: 1-(1-(1-1/Ny^(k(k-1)/2))^t)
Plotting this formula on two graphs for two different Ny population sizes.
Each line gives the probabilities all k number of lineages for one given number of generations ago. For example, for Ny = 50, the probability that any of 16 lineages coalesce in the previous generation (t = 1) is over 90%. See here again that for larger population sizes, the probabilities are lower given the same values for t and k See also how for larger k and t values, the probabilities of coalescence are higher.
The most important thing to note here is that, when k values are high, you don’t need a high t value for a high probability of coalescence. Meaning, many lineages coalesce only a few generations ago. However, when k values are low, then large values of t are needed for a descent probability of coalescence. Meaning, deep branches are more likely to coalesce many generations ago. Both of these effects means that coalescent trees are top heavy; i.e. many branches coalesce at the tips, deep branches coalesce at the base.
For example, in graph Ny = 50, the probability of coalescence among 8 random lineages 1 or 2 generations ago is about 50%. Meaning, with a 50% probability, we would expect to see 8 lineages to still be separate 1 to 2 generation ago within a population of 50. The probability of 10 separate lineages is lower because the probability of coalescence is 60%, and we certainly would not expect to see 16 lineage to all remain separate 1 generation ago as the probability of coalescence is 90%. So, with a 50% probability, 16 lineages would have coalesced down to about 8 lineages 2 generations ago. Many branches coalesce at the top. If we go further, the probability of coalescence among 4 lineages reaches 50% between 5 to 8 generations ago. Regarding 3 lineages, 50% coalescence probability is reached 12 generations ago. However, for 2 lineages to coalesce in a population of 50, you would need to go back between 30 to 40 generations ago to get a probability of 50%. Deep branches coalesce at the base. We can do the same for the other population size Ny =300, although here we need more generations to reach the same branch points. We would still expect to see about 15 lineages to be separate 1 generation ago. 8 lineages would remain separate between 5 and 8 generations ago. 4 lineages remain separate until 30 to 40 generations ago. And the last 2 lineages would remain separate until over 200 generations ago.
Thus, each population size, we would expect to something like the following coalescent trees:
Note here that the number of branches increases almost exponentially the closer you get to the present. Yet, we have only considered constant population sizes so far. So, it is already clear that Jeanson is wrong to say that an exponential increase in the number of branches corresponds to an exponential population growth. But we can go further and see how a coalescent tree would look like if the population is exponentially growing. We can calculate the population size at t by the formula: Ny0(1+r)^-t with Ny0 defined as the population size at t=0, and r is the growth factor. Let’s have Ny0 = 2000 and use exponential growth rates of 0,01 and 0,04. We get the following in graph form:
Now the probabilities look very different. I did not bother the include the lines for t = 2 to 9 since the probability of coalescence to occur among 16 lineages within 10 generations ago is <50%. This is because among these generations, the population sizes are still quite large. Coalescence are more likely to occur further in the past when population sizes were small. If we construct the expected coalescence trees for these like last time, we get the following.
Very different. Deep branches are more likely to coalesce more recently and recent branches are more likely to coalesce further into the past when the population is growing exponentially. Maybe the difference isn’t as obvious. Let’s take a very large population size (8 billion) and a large growth rate (10%). Then we get the following:
Most branches coalesce many generations ago when the population size was small. This is not what Jeanson expects from an exponential population growth. He wrongly thinks that more branching is an indication that a population is growing. BUT we have seen that a greater frequency of branching events (i.e. coalescence) correlates with smaller population sizes, NOT with a growing population. The way we actually use coalescence to reconstruct population size in the past is by examining time when coalescence rates are high (smaller population sizes, perhaps bottle necks) and when the rates were low (when the population sizes were larger). There have been such studies before, and those use a lot more complicated math and models that take things into account such as non-random mating that I did not previously with my simple calculations.
Here is an example for how mtDNA variation predicts population size. The outlier here is Australia+New Guinea and they discuss the implications in the paper.
To be honest I skimmed the end - a lot of info…
