ROOT PROBABILITIES FOR INTRASPECIFIC GENE TREES UNDER NEUTRAL COALESCENT THEORY

The rooting of intraspecific gene or haplotype tree has proven to be difficult using the traditional techniques of rooting species trees such as outgroups. As an alternative, we apply neutral coalescent theory to the problems of determining the root of an intraspecific gene tree and the relative ages of the haplotypes. By using a recursive equation, exact root probabilities can be calculated for small cladograms and sample sizes. These exact probabilities indicate that root probabilities are not very sensitive to the parameter theta, which is four times the mutation rate times the inbreeding effective size. The exact probabilities also indicate that root probabilities are not very sensitive to the absolute numbers of each haplotype (allele) class, only their relative proportions. The exact method, unfortunately, is not feasible for implementation with larger data sets with present algorithms and computers. However, the exact results suggested a simple heuristic for determining outgroup weights; that is, finding the haplotype that is the oldest in the sample and that can serve as an outgroup for the remainder of the haplotype tree. Computer simulations revealed that these outgroup weights are strongly correlated with actual age and are much better indicators of haplotype age than is the haplotype frequency, another commonly used indicator of relative age. (C) 1994 Academic Press, Inc.