Some statistical issues in population genetics

Abstract

We wish to report some improvements to existing statistical methodology that uses dominant marker data to infer population genetic structure. We first show that our proposed zero-correction procedure reduces the root mean square error of candidate estimators of locus-specific null allele frequency and heterozygosity. Next, we demonstrate how correction for ascertainment bias in the estimation of average heterozygosity can be done, using a linear transformation of the sample average heterozygosity. Subsequently, we propose two ways of evaluating the maximum likelihood estimator of average heterozygosity in a single population: one using the truncated beta-binomial likelihood, and another using the EM algorithm. Finally, using a simulation approach, we argue that the categorical analysis of variance (CATANOVA) framework, instead of the commonly used analysis of molecular variance (AMOVA), is the appropriate one for analysing genetic structure in a collection of populations, where interest is intrinsically centered on the latter.