Models and partition of variance for quantitative trait loci with epistasis and linkage disequilibrium

Abstract

Background

A genetic model about quantitative trait loci (QTL) provides a basis to interpret the genetic basis of quantitative traits in a study population, such as additive, dominance and epistatic effects of QTL and the partition of genetic variance. The standard quantitative genetics model is based on the least squares partition of genetic effects and also genetic variance in an equilibrium population. However, over years many specialized QTL models have also been proposed for applications in some specific populations. How are these models related? How to analyze and partition a QTL model and genetic variance when both epistasis and linkage disequilibrium are considered?

Results

Starting from the classical description of Cockerham genetic model, we first represent the model in a multiple regression setting by using indicator variables to describe the segregation of QTL alleles. In this setting, the definition of additive, dominance and epistatic effects of QTL and the basis for the partition of genetic variance are elaborated. We then build the connection between this general genetic model and a few specialized models (a haploid model, a diploid F₂ model and a general two-allele model), and derive the genetic effects and partition of genetic variance for multiple QTL with epistasis and linkage disequilibrium for these specialized models.

Conclusion

In this paper, we study extensively the composition and property of the genetic model parameters, such as genetic effects and partition of genetic variance, when both epistasis and linkage disequilibrium are considered. This is the first time that both epistasis and linkage disequilibrium are considered in modeling multiple QTL. This analysis would help us to understand the structure of genetic parameters and relationship of various genetic quantities, such as allelic frequencies and linkage disequilibrium, on the definition of genetic effects, and will also help us to understand and properly interpret estimates of the genetic effects and variance components in a QTL mapping experiment.

Background

Modeling quantitative trait loci (QTL) started with Yule [1, 2] and Pearson [3] (see [4, 5] for the early history of quantitative genetics). However, it was Fisher [6] who laid the firm foundation for quantitative genetics. Fisher defined gene effects (additive, dominance and epistatic effects) based on the partition of genetic variance. He partitioned the genetic variance into a portion due to additive effects (averaged allelic substitution effects), a portion due to dominance effects (allelic interactions), and a portion due to epistatic effects (non-allelic interactions) of genes. He then studied the correlation between relatives using the model. Cockerham [7] used the orthogonal contrasts to redefine the additive and dominance effects of QTL and, by extending the contrasts to include epistatic effects, he partitioned the epistatic variance of two loci into those due to additive × additive, additive × dominance, dominance × additive and dominance × dominance effects of QTL. Cockerham then generalized the model to multiple loci. This was further generalized by Kempthorne [8, 9] to multiple alleles. This model has been used as the basis for studying quantitative genetics ever since.

However, over years, many specialized models have also been proposed. Some are just special cases of the general genetic model and some are simplified variants tailored for particular applications or interpretations. With the propagation of numerous quantitative genetic models, there have also been some confusions in literature on the definition and interpretation of additive, dominance and epistatic effects of QTL and their relationship to the partition of genetic variance. Also, there has never been a study that considers both epistasis and linkage disequilibrium in the partition of genetic variance for multiple QTL. In this paper, we try to build the connection between the general genetic model and a few other commonly used genetic models to clarify the basis for the interpretation of different genetic models.

We start with an introduction of the genetic model as expressed in [7] in the context of variance components. Then by introducing an indicator variable for each QTL allele, we represent the model in a multiple regression setting and examine the definition and meaning of the genetic effects (additive, dominance and epistatic effects) of QTL and partition of the genetic variance in an equilibrium population and also in a disequilibrium population. Most of previous studies on modeling QTL discuss epistasis only in reference to an equilibrium population. An examination of properties of a model with both epistasis and linkage disequilibrium is important for QTL analysis in both experimental and natural populations. This is another goal of the paper and is studied in great detail here. We discuss a few reduced models used for QTL analysis, such as backcross model (essentially a haploid model) and F₂ model. We also give details for a general two-allele model which may be useful for studying the genetic architecture in a natural population using single nucleotide polymorphisms (SNPs).

Previously, in [10], we compared F₂ model and the general two-allele model with another commonly used genetic model, called F_∞ model. By specifying the basis of definition for each model, we compared the properties of these models in the estimation and interpretation of QTL effects including epistasis and discussed a few potential problems of using F_∞ model in a segregating population for QTL analysis. Similarly, we also compared these models with another model proposed by Cheverud [11, 12].

An important result of [10] is that the genetic effects defined in reference to an equilibrium population also apply to a disequilibrium population. The partial regression coefficients, that define the genetic effects in a disequilibrium population, equal to the simple regression coefficients in a corresponding equilibrium population – the usual basis to define and interpret a genetic effect including an epistatic effect. Hardy-Weinberg and linkage disequilibria only introduce covariances between different genetic effects. With this result, in this paper our discussion on epistasis and linkage disequilibrium is focused on the partition and composition of genetic variances and covariances between different genetic effects in different populations.

Results

The genetic model

A general genetic model for the partition of genetic variance (particularly epistatic variance) in a random mating population was first given by Cockerham [7, 13] and extended to multiple alleles by Kempthorne [8, 9], following the basic genetic model formulated by Fisher [6]. The model for two loci A and B with multiple alleles was expressed as follows

$\begin{array}{c}{G}_{jl}^{ik}=\mu +{\alpha }^i+{\alpha }_j+{\delta }_j^i+{\beta }^k+{\beta }_l+{\gamma }_l^k+({\alpha }^i{\beta }^k)\\ +({\alpha }^i{\beta }_l)+({\alpha }_j{\beta }^k)+({\alpha }_j{\beta }_l)+({\alpha }^i{\gamma }_l^k)\\ +({\alpha }_j{\gamma }_l^k)+({\delta }_j^i{\beta }^k)+({\delta }_j^i{\beta }_l)+({\delta }_j^i{\gamma }_l^k)\end{array}\left(1\right)$

where the genotypic value $G_{jl}^{ik}$ is the expected phenotype of an individual carrying alleles A _i , A _j , B _k , and B _l with phased genotype A _i B _k /A _j B _l formed by the union of a paternal gamete A _i B _k and a maternal gamete A _j B _l . The model partitions the total genotypic value into a number of genetic effects which include additive effects of each allele (α's and β's), dominance effects between two alleles at each locus (δ's and γ's), additive × additive interactions between two alleles at two loci ((αβ)'s), additive × dominance interactions involving three alleles ((αγ)'s and (δβ)'s), and dominance × dominance interaction involving all four alleles ((δγ)'s).

As an ANOVA model, it is known that not all the parameters in model (1) are estimable. A number of constraint conditions on these parameters are therefore needed. Let pⁱ, q^kdenote allelic frequencies for alleles on paternal gametes, and p _j , q _l allelic frequencies for alleles on maternal gametes. It is usually assumed that a weighted summation of genetic effects is zero over any index for each genetic component as a deviation from the mean. Some examples are

$\begin{array}{l}{\displaystyle \sum _i{p}^i{\alpha }^i=0,{\displaystyle \sum _i{p}^i{\delta }_j^i=0,{\displaystyle \sum _j{p}_j{\delta }_j^i=0,}}}\hfill \\ {\displaystyle \sum _i{p}^i({\alpha }^i{\beta }^k)=0,}{\displaystyle \sum _k{p}^k({\alpha }^i{\beta }^k)=0,}\hfill \\ {\displaystyle \sum _j{p}_j({\delta }_j^i{\beta }^k)=0,{\displaystyle \sum _k{q}^k({\delta }_j^i{\beta }^k)=0,}}\cdots \cdots \hfill \end{array}\left(2\right)$

Under the assumption of random mating and linkage equilibrium and allowing for different allelic frequencies in paternal and maternal gametes, the mean and genetic effects can be expressed as follows based on the least squares principle:

$\begin{array}{l}\mu ={\displaystyle \sum _{i,j,k,l}{p}^i{p}_j{q}^k{q}_l{G}_{jl}^{ik}}\left(3\right)\\ {\alpha }^i=G_{\mathrm{..}}^{i.}-G_{\mathrm{..}}^{\mathrm{..}},{\beta }^k=G_{\mathrm{..}}^{.k}-G_{\mathrm{..}}^{\mathrm{..}},\\ {\alpha }_j=G_{j.}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}},{\beta }_l=G_{.l}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}},\\ {\delta }_j^i=G_{j.}^{i.}-G_{\mathrm{..}}^{i.}-G_{j.}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}},\\ {\gamma }_l^k=G_{.l}^{.k}-G_{\mathrm{..}}^{.k}-G_{.l}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}},\\ ({\alpha }^i{\beta }^k)=G_{\mathrm{..}}^{ik}-G_{\mathrm{..}}^{i.}-G_{\mathrm{..}}^{.k}+G_{\mathrm{..}}^{\mathrm{..}},\\ ({\alpha }^i{\beta }_l)=G_{.l}^{i.}-G_{\mathrm{..}}^{i.}-G_{.l}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}},\\ ({\alpha }_j{\beta }^k)=G_{j.}^{.k}-G_{j.}^{\mathrm{..}}-G_{\mathrm{..}}^{.k}+G_{\mathrm{..}}^{\mathrm{..}},\\ ({\alpha }_j{\beta }_l)=G_{jl}^{\mathrm{..}}-G_{j.}^{\mathrm{..}}-G_{.l}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}},\\ ({\alpha }^i{\gamma }_l^k)=G_{.l}^{ik}-G_{\mathrm{..}}^{ik}-G_{.l}^{i.}-G_{.l}^{.k}+G_{\mathrm{..}}^{i.}+G_{\mathrm{..}}^{.k}\\ +G_{.l}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}},\\ ({\alpha }_j{\gamma }_l^k)=G_{jl}^{.k}-G_{j.}^{.k}-G_{.l}^{.k}-G_{jl}^{\mathrm{..}}+G_{j.}^{\mathrm{..}}+G_{\mathrm{..}}^{.k}\\ +G_{.l}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}},\\ ({\delta }_j^i{\beta }^k)=G_{j.}^{ik}-G_{\mathrm{..}}^{ik}-G_{j.}^{i.}-G_{j.}^{.k}+G_{\mathrm{..}}^{i.}+G_{\mathrm{..}}^{.k}\\ +G_{j.}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}},\\ ({\delta }_j^i{\beta }_l)=G_{jl}^{i.}-G_{j.}^{i.}-G_{.l}^{i.}-G_{jl}^{\mathrm{..}}+G_{\mathrm{..}}^{i.}+G_{j.}^{\mathrm{..}}\\ +G_{.l}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}},\\ ({\delta }_j^i{\gamma }_l^k)=G_{jl}^{ik}-G_{j.}^{ik}-G_{.l}^{ik}-G_{jl}^{i.}+G_{jl}^{.k}+G_{\mathrm{..}}^{ik}\\ +G_{jl}^{\mathrm{..}}+G_{j.}^{i.}+G_{.l}^{.k}+G_{j.}^{.k}+G_{.l}^{i.}-G_{\mathrm{..}}^{i.}\\ -G_{\mathrm{..}}^{.k}-G_{j.}^{\mathrm{..}}-G_{.j}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}}\end{array}$

where $G_{\mathrm{..}}^{\mathrm{..}}={\displaystyle {\sum }_{i,j,k,l}{p}^i{p}_j{q}^k{q}_l{G}_{jl}^{ik}}$, $G_{\mathrm{..}}^{i.}={\displaystyle {\sum }_{j,k,l}{p}_j{q}^k{q}_l{G}_{jl}^{ik}}$, and so on. The total genetic variance is $V_G={\displaystyle {\sum }_{i,j,k,l}{p}^i{p}_j{p}^k{p}_l{(G_{jl}^{ik}-\mu )}^2}$, and has an orthogonal partition under random mating and linkage equilibrium

$\begin{array}{l}{V}_G=V_{A_1}+V_{A_2}+V_{D_1}+V_{D_2}+V_{A_1{A}_2}+V_{A_1{D}_2}\left(4\right)\\ +V_{D_1{A}_2}+V_{D_1{D}_2}\end{array}$

with

$\begin{array}{l}{V}_{A_1}={\displaystyle \sum _i{p}^i{({\alpha }^i)}^2}+{\displaystyle \sum _j{p}_j{({\alpha }_j)}^2}\\ V_{D_1}={\displaystyle \sum _{i,j}{p}^i{p}_j{({\delta }_j^i)}^2}\\ V_{A_2}={\displaystyle \sum _k{q}^k{({\beta }^k)}^2}+{\displaystyle \sum _l{q}_l{({\beta }_l)}^2}\\ V_{D_2}={\displaystyle \sum _{k,l}{q}^k{q}_l{({\gamma }_l^k)}^2}\\ V_{A_1{A}_2}={\displaystyle \sum _{i,k}{p}^i{q}^k{({\alpha }^i{\beta }^k)}^2}+{\displaystyle \sum _{j,l}{p}_j{q}_l{({\alpha }_j{\beta }_l)}^2}\\ +{\displaystyle \sum _{i,l}{p}^i{q}_l{({\alpha }^i{\beta }_l)}^2}+{\displaystyle \sum _{j,k}{p}_j{q}^k{({\alpha }_j{\beta }^k)}^2}\\ V_{A_1{D}_2}={\displaystyle \sum _{i,k,l}{p}^i{q}^k{q}_l{({\alpha }^i{\gamma }_l^k)}^2}+{\displaystyle \sum _{j,k,l}{p}_j{q}^k{q}_l{({\alpha }_j{\gamma }_l^k)}^2}\\ V_{D_1{A}_2}={\displaystyle \sum _{i,j,k}{p}^i{p}_j{q}^k{({\delta }_j^i{\beta }^k)}^2}+{\displaystyle \sum _{i,j,l}{p}^i{p}_j{q}_l{({\delta }_j^i{\beta }_l)}^2}\\ V_{D_1{D}_2}={\displaystyle \sum _{i,j,k,l}{p}^i{p}_j{q}^k{q}_l{({\delta }_j^i{\gamma }_l^k)}^2}\end{array}$

Using indicator variables, we can represent model (1) in another form. Assume that the two loci A and B have alleles A _i , i = 1, 2, ..., n₁; and B _k , i = 1, 2, ..., n₂, respectively. We define the following indicator variables to represent the segregation of alleles in a population.

$\begin{array}{l}{z}_{M_i}^{(1)}=\{\begin{array}{ll}1,\hfill & \text{for }{A}_i\text{ allele from paternal gamete}\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}\\ z_{F_j}^{(1)}=\{\begin{array}{ll}1,\hfill & \text{for }{A}_j\text{ allele from maternal gamete}\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}\end{array}$

for i, j = 1, 2, ..., n₁ at locus A, and

$\begin{array}{l}{z}_{M_k}^{(2)}=\{\begin{array}{ll}1,\hfill & \text{for }{B}_k\text{ allele from paternal gamete}\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}\\ z_{F_l}^{(2)}=\{\begin{array}{ll}1,\hfill & \text{for }{B}_l\text{ allele from maternal gamete}\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}\end{array}$

for k, l = 1, 2, ..., n₂ at locus B. In terms of these indicator variables, we have the following.

• Hardy-Weinberg equilibrium (HWE) implies that {$z_{M_i}^{(1)}$, i = 1, 2, ..., n₁} are independent of {$z_{F_j}^{(1)}$, j = 1, 2, ..., n₁}, and {$z_{M_k}^{(2)}$, k = 1, 2, ..., n₂} are independent of {$z_{F_l}^{(2)}$, l = 1, 2, ..., n₂}.

• Linkage equilibrium (LE) implies that {$z_{M_i}^{(1)}$, i = 1, 2, ..., n₁} are independent of {$z_{M_k}^{(2)}$, k = 1, 2, ..., n₂}, and {$z_{F_j}^{(1)}$, j = 1, 2, ..., n₁} are independent of {$z_{F_l}^{(2)}$, l = 1, 2, ..., n₂}.

• There is another type of disequilibrium; i.e., the so-called genotypic disequilibrium [14] for two alleles on different gametes and at different loci. So, the genotypic equilibrium (GE) here means that {$z_{M_i}^{(1)}$, i = 1, 2,..., n₁} are independent of {$z_{F_l}^{(2)}$, l = 1, 2, ..., n₂}, and {$z_{F_j}^{(1)}$, j = 1, 2, ..., n₁} are independent of {$z_{M_k}^{(2)}$, k = 1, 2, ..., n₂}.

It is known that under random mating we have both HWE and GE, which together are called gametic phase equilibrium. Now, let G denote the genotypic value of a progeny drawn randomly from the current population. Based on Cockerham model, G can be expressed as

$\begin{array}{l}G=\mu +{\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{z}_{M_i}^{(1)}}+{\displaystyle \sum _{j=1}^{n_1}{\alpha }_j{z}_{F_j}^{(1)}}+{\displaystyle \sum _{i,j}{\delta }_j^i{z}_{M_i}^{(1)}{z}_{F_j}^{(1)}}\\ +{\displaystyle \sum _{k=1}^{n_2}{\beta }^k{z}_{M_k}^{(2)}}+{\displaystyle \sum _{l=1}^{n_2}{\beta }_l{z}_{F_l}^{(2)}}+{\displaystyle \sum _{k,l}{\gamma }_l^k{z}_{M_k}^{(2)}{z}_{F_l}^{(2)}}\\ +[{\displaystyle \sum _{i,k}({\alpha }^i{\beta }^k)z_{M_i}^{(1)}{z}_{M_k}^{(2)}}+{\displaystyle \sum _{i,l}({\alpha }^i{\beta }_l)z_{M_i}^{(1)}{z}_{F_l}^{(2)}}\\ +{\displaystyle \sum _{j,k}({\alpha }_j{\beta }^k)z_{F_j}^{(1)}{z}_{M_k}^{(2)}}+{\displaystyle \sum _{j,l}({\alpha }_j{\beta }_l)z_{F_j}^{(1)}{z}_{F_l}^{(2)}}]\\ +[{\displaystyle \sum _{i,k,l}({\alpha }^i{\gamma }_l^k)z_{M_i}^{(1)}{z}_{M_k}^{(2)}{z}_{F_l}^{(2)}}+{\displaystyle \sum _{j,k,l}({\alpha }_j{\gamma }_l^k)z_{F_j}^{(1)}{z}_{M_k}^{(2)}{z}_{F_l}^{(2)}}]\\ +[{\displaystyle \sum _{i,j,k}({\delta }_j^i{\beta }^k)z_{M_i}^{(1)}{z}_{F_j}^{(1)}{z}_{M_k}^{(2)}}+{\displaystyle \sum _{i,j,l}({\delta }_j^i{\beta }_l)z_{M_i}^{(1)}{z}_{F_j}^{(1)}{z}_{F_l}^{(2)}}]\\ +{\displaystyle \sum _{i,j,k,l}({\delta }_j^i{\gamma }_l^k)z_{M_i}^{(1)}{z}_{F_j}^{(1)}{z}_{M_k}^{(2)}{z}_{F_l}^{(2)}}\left(5\right)\end{array}$

This is simply a different presentation of Cockerham model with the same constraint conditions applied on the coefficient parameters. For a given individual with genotype A _i B _k /A _j B _l , G will take the same value of $G_{jl}^{ik}$ as before. However, this expression is helpful for us to understand some details about each component of genetic effects. We can see this more clearly in the examination of some reduced models later.

In general, the genetic effects can be defined separately for alleles that are paternally and maternally transmitted to account for possible biological differences. As a fully parameterized model for $G_{jl}^{ik}$, model (3) may give $G_{jl}^{ik}\ne G_{jk}^{il}\ne G_{il}^{jk}\ne G_{ik}^{jl}$ depending on how genetic effects are defined. If locus A has n₁ alleles, and locus B has n₂ alleles, there are N = $n_1^2{n}_2^2$ possible phased genotypes in total with the partition of the degrees of freedom given in Table 1.

Table 1 Partition of degrees of freedom for two loci with number of alleles n₁ and n₂ (a general case)

If we assume that the union of paternal gamete A _i B _k with maternal gamete A _j B _l have the same mean effect as that of paternal gamete A _j B _l with maternal gamete A _i B _k (i.e., $G_{jl}^{ik}=G_{ik}^{jl}$), the coupling and repulsion heterozygotes have the same genotypic value (i.e., $G_{jl}^{ik}=G_{jk}^{il}$), and paternal and maternal gametes have the same gametic frequency distribution, we do not need to distinguish paternal and maternal effects. In this case, the two loci can be regarded as 2 factors and each factor has $n_i^2$ (i = 1, 2) levels produced by the allelic combinations of n _i alleles (cf. [7]). The total number of genotypes is N = n₁(n₁ + 1)n₂(n₂ + 1)/4 and the partition of degrees of freedom is shown in Table 2. Since in this case, αⁱ= α _i , β^k= β _k , ..., and so on, the model can also be expressed as follows

$\begin{array}{l}G=\mu +{\displaystyle \sum _{i=1}^{n_1}{\alpha }_i(z_{M_i}^{(1)}+z_{F_i}^{(1)})}+{\displaystyle \sum _{i,j}{\delta }_j^i{z}_{M_i}^{(1)}{z}_{F_j}^{(1)}}\\ +{\displaystyle \sum _{k=1}^{n_2}{\beta }_k(z_{M_k}^{(2)}+z_{F_k}^{(2)})}+{\displaystyle \sum _{k,l}{\gamma }_l^k{z}_{M_k}^{(2)}{z}_{F_l}^{(2)}}\\ +{\displaystyle \sum _{i,k}({\alpha }_i{\beta }_k)}(z_{M_i}^{(1)}+z_{F_i}^{(1)})(z_{M_k}^{(2)}+z_{F_k}^{(2)})\\ +{\displaystyle \sum _{i,k,l}({\alpha }_i{\gamma }_l^k)}(z_{M_i}^{(1)}+z_{F_i}^{(1)})z_{M_k}^{(2)}{z}_{F_l}^{(2)}\\ +{\displaystyle \sum _{i,j,k}({\delta }_j^i{\beta }_k)}{z}_{M_i}^{(1)}{z}_{F_j}^{(1)}(z_{M_k}^{(2)}+z_{F_k}^{(2)})\\ +{\displaystyle \sum _{i,j,k,l}({\delta }_j^i{\gamma }_l^k)}{z}_{M_i}^{(1)}{z}_{F_j}^{(1)}{z}_{M_k}^{(2)}{z}_{F_l}^{(2)})\left(6\right)\end{array}$

Table 2 Partition of degrees of freedom for two loci with number of alleles n₁ and n₂ (a simplified case without distinguishing the paternal and maternal origins)

For the case of an arbitrary number of loci, the situation will become more complicated. In addition to the additive and dominance effects at each locus and two locus interactions (additive × additive, additive × dominance, dominance × additive, dominance × dominance, with a total number of 2² terms), there are 3 locus interactions (additive × additive × additive, additive × additive × dominance, ..., with a total number of 2³ terms), 4 locus interactions (additive × additive × additive × additive, ..., with a total number of 2⁴ terms), and so on. Though the extension is straightforward, the total number of terms will increase dramatically. We will show some models with multiple loci in later examples by ignoring trigenic and higher order epistasis.

Effects and variance components

Let pⁱ, p _j (i, j = 1, 2, ..., n₁) be allelic frequencies of paternal and maternal gametes at locus A, respectively. Let also q^k, q _l (k, l = 1, 2, ..., n₂) denote allelic frequencies of paternal and maternal gametes at locus B, respectively. In the analysis of variance for the model, it is convenient to use deviations of the indicator variables $z_{M_i}^{(1)}$, $z_{F_i}^{(1)}$, $z_{M_j}^{(2)}$ and $z_{F_j}^{(2)}$ from their expected values. That is

$\begin{array}{l}{x}_{M_i}^{(1)}=z_{M_i}^{(1)}-E(z_{M_i}^{(1)})=z_{M_i}^{(1)}-p^i\\ =\{\begin{array}{ll}1-p^i,\hfill & \text{for }{A}_i\text{ allele from paternal gamete}\hfill \\ -p^i,\hfill & \text{otherwise}\hfill \end{array}\end{array}$

Similarly, define

$\begin{array}{l}{x}_{F_j}^{(1)}=z_{F_j}^{(1)}-E(z_{F_j}^{(1)})=z_{F_j}^{(1)}-p_j\\ x_{M_k}^{(2)}=z_{M_k}^{(2)}-E(z_{M_k}^{(2)})=z_{M_k}^{(2)}-q^k\\ x_{F_l}^{(2)}=z_{F_l}^{(2)}-E(z_{F_l}^{(2)})=z_{F_l}^{(2)}-q_l\end{array}$

Taking the constraint conditions on the genetic effects into account, we can show that,

$\begin{array}{c}{\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{x}_{M_i}^{(1)}}={\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{z}_{M_i}^{(1)}}\\ {\displaystyle \sum _{i,j}{\delta }_j^i{x}_{M_i}^{(1)}{x}_{F_j}^{(1)}}={\displaystyle \sum _{i,j}{\delta }_j^i{z}_{M_i}^{(1)}{z}_{F_j}^{(1)}}\\ {\displaystyle \sum _{i,k,l}({\alpha }^i{\gamma }_l^k)x_{M_i}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}}={\displaystyle \sum _{i,k,l}({\alpha }^i{\gamma }_l^k)z_{M_i}^{(1)}{z}_{M_k}^{(2)}{z}_{F_l}^{(2)}}\\ {\displaystyle \sum _{i,j,k,l}({\delta }_j^i{\gamma }_l^k)x_{M_i}^{(1)}{x}_{F_j}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}}={\displaystyle \sum _{i,j,k,l}({\delta }_j^i{\gamma }_l^k)z_{M_i}^{(1)}{z}_{F_j}^{(1)}{z}_{M_k}^{(2)}{z}_{F_l}^{(2)}}\end{array}$

and so on. For example,

$\begin{array}{l}{\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{x}_{M_i}^{(1)}}={\displaystyle \sum _{i=1}^{n_1}{\alpha }^i(z_{M_i}^{(1)}-p^i)}\\ ={\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{z}_{M_i}^{(1)}}-{\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{p}^i={\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{z}_{M_i}^{(1)}}}\end{array}$

as ${\displaystyle {\sum }_{i=1}^{n_1}{\alpha }^i{p}^i}=0$ by the constrain condition (2). Therefore, model (5) can be rewritten as

$\begin{array}{l}G=\mu +{\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{x}_{M_i}^{(1)}}+{\displaystyle \sum _{j=1}^{n_1}{\alpha }^i{x}_{F_j}^{(1)}}+{\displaystyle \sum _{i,j}{\delta }_j^i{x}_{M_i}^{(1)}{x}_{F_j}^{(1)}}\\ +{\displaystyle \sum _{k=1}^{n_2}{\beta }^k{x}_{M_k}^{(2)}}+{\displaystyle \sum _{l=1}^{n_2}{\beta }_l{x}_{F_l}^{(2)}}+{\displaystyle \sum _{k,l}{\gamma }_l^k{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}}\\ +[{\displaystyle \sum _{i,k}({\alpha }^i{\beta }^k)x_{M_i}^{(1)}{x}_{M_k}^{(2)}}+{\displaystyle \sum _{i,l}({\alpha }^i{\beta }_l)x_{M_i}^{(1)}{x}_{F_l}^{(2)}}\\ +{\displaystyle \sum _{j,k}({\alpha }_j{\beta }^k)x_{F_j}^{(1)}{x}_{M_k}^{(2)}}+{\displaystyle \sum _{j,l}({\alpha }_j{\beta }_l)x_{F_j}^{(1)}{x}_{F_l}^{(2)}]}\\ +[{\displaystyle \sum _{i,k,l}({\alpha }^i{\gamma }_l^k)x_{M_i}^{(1)}{x}_{M_k}^{(2)}}{x}_{F_l}^{(2)}+{\displaystyle \sum _{j,k,l}({\alpha }_j{\gamma }_l^k)x_{F_j}^{(1)}{x}_{M_k}^{(2)}}{x}_{F_l}^{(2)}]\\ +[{\displaystyle \sum _{i,j,k}({\delta }_j^i{\beta }^k)x_{M_i}^{(1)}{x}_{F_j}^{(1)}}{x}_{M_k}^{(2)}+{\displaystyle \sum _{i,j,l}({\delta }_j^i{\beta }_l)x_{M_i}^{(1)}{x}_{F_j}^{(1)}}{x}_{F_l}^{(2)}]\\ +{\displaystyle \sum _{i,j,k,l}({\delta }_j^i{\gamma }_l^k)x_{M_i}^{(1)}{x}_{F_j}^{(1)}}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}\left(7\right)\end{array}$

When we deal with some reduced models in the next section, we will find that the model of this form is especially helpful as it makes the model parameter constraints built into regression variables which is suited for genetic interpretation. The form of model (7) can also facilitate the demonstration that under Hardy-Weinberg, linkage and genotypic equilibria, the regression coefficients (genetic effects) are Cockerham's least squares effects (3) (Appendix A), and the genotypic variance V _G has the orthogonal partition (4) (Appendix B).

Now we discuss the properties of model in a disequilibrium situation. As stated in [14], there are three types of disequilibria

• Typel: between alleles on the same gametes but at different loci

• Type2: between alleles at the same locus but on different gametes

• Type3: between alleles on different gametes and at different loci.

If we denote $P_{jl}^{ik}$ as the genotypic frequency of A _i B _k /A _j B _l , $P_{j.}^{i.}$ as the genotypic frequency of A _i /A _j , and so on, following [14], the digenic disequilibria can be written as

$\begin{array}{l}{D}_{\mathrm{..}}^{ik}=\text{Cov}(z_{M_i}^{(1)},z_{M_k}^{(2)})=E(x_{M_i}^{(1)}{x}_{M_k}^{(2)})=P_{\mathrm{..}}^{ik}-p^i{q}^k\\ D_{jl}^{\mathrm{..}}=\text{Cov}(z_{F_j}^{(1)},z_{F_l}^{(2)})=E(x_{F_j}^{(1)}{x}_{F_l}^{(2)})=P_{jl}^{\mathrm{..}}-p_j{q}_l\\ D_{j.}^{i.}=E(x_{M_i}^{(1)}{x}_{F_j}^{(1)})=P_{j.}^{i.}-p^i{p}_j\\ D_{.l}^{.k}=E(x_{M_k}^{(2)}{x}_{F_l}^{(2)})=P_{.l}^{.k}-q^k{q}_l\\ D_{.j}^{i.}=E(x_{M_i}^{(1)}{x}_{F_l}^{(2)})=P_{.l}^{i.}-p^i{q}_l\\ D_{j.}^{.k}=E(x_{F_j}^{(1)}{x}_{M_k}^{(2)})=P_{j.}^{.k}-p_j{q}^k.\end{array}$

And the trigenic disequilibria

$\begin{array}{c}{D}_{j.}^{ik}=E(x_{M_i}^{(1)}{x}_{F_j}^{(1)}{x}_{M_k}^{(2)})\\ =P_{j.}^{ik}-p^i{P}_{j.}^{.k}-q^k{P}_{j.}^{i.}-p_j{P}_{\mathrm{..}}^{ik}+2p^i{p}_j{q}^k\\ D_{.l}^{ik}=E(x_{M_i}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)})\\ =P_{.l}^{ik}-p^i{P}_{.l}^{.k}-q^k{P}_{.l}^{i.}-q_l{P}_{\mathrm{..}}^{ik}+2p^i{q}^k{q}_l\\ D_{jl}^{i.}=E(x_{M_i}^{(1)}{x}_{F_j}^{(1)}{x}_{F_l}^{(2)})\\ =P_{jl}^{i.}-p^i{P}_{jl}^{\mathrm{..}}-p_j{P}_{\mathrm{..}}^{i.}-q_l{P}_{j.}^{i.}+2p^i{p}_j{q}_l\\ D_{jl}^{.k}=E(x_{F_j}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)})\\ =P_{jl}^{.k}-p_j{P}_{.l}^{.k}-q^k{P}_{j.}^{.l}-q_l{P}_{j.}^{.k}+2p_j{q}^k{q}_l\end{array}$

Similarly for the quadrigenic disequilibrium, we may define

$\begin{array}{c}{D}_{jl}^{ik}=E(x_{M_i}^{(1)}{x}_{F_j}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)})\\ =P_{jl}^{ik}-p^i{P}_{jl}^{.k}-p_j{P}_{.l}^{ik}-q^k{P}_{jl}^{i.}-q_l{P}_{j.}^{ik}\\ +p^i{p}_j{P}_{.l}^{.k}-p^i{q}^k{P}_{jl}^{\mathrm{..}}+p^i{q}_l{P}_{j.}^{.k}+p_j{q}^k{P}_{.l}^{i.}\\ +p_j{q}_l{P}_{\mathrm{..}}^{ik}+q^k{q}_l{P}_{j.}^{i.}-3p^i{p}_j{q}^k{q}_l.\end{array}$

If we express $D_{jl}^{ik}$ as a function of lower-order linkage disequilibria, we have

$\begin{array}{c}{D}_{jl}^{ik}=P_{jl}^{ik}-p^i{D}_{jl}^{.k}-p_j{D}_{.l}^{ik}-q^k{D}_{jl}^{i.}-q_l{D}_{j.}^{ik}\\ -p^i{p}_j{D}_{.l}^{.k}-p^i{q}^k{D}_{jl}^{\mathrm{..}}-p^i{q}_l{D}_{j.}^{.k}-p_j{q}^k{D}_{.l}^{i.}\\ -p_j{q}_l{D}_{\mathrm{..}}^{ik}-q^k{q}_l{D}_{j.}^{i.}-p^i{p}_j{q}^k{q}_l.\end{array}$

This definition is the same as that given by [15, 16]. Note that ${\displaystyle {\sum }_i{z}_{M_i}^{(1)}}=1$. Then, we have

$\begin{array}{l}{\displaystyle \sum _i{D}_{jl}^{ik}}=E[({\displaystyle \sum _i{x}_{M_i}^{(1)})x_{F_j}^{(1)}}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}]\\ =E[{\displaystyle \sum _i(z_{M_i}^{(1)}-p^i)x_{F_j}^{(1)}}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}=0\end{array}$

In general, $D_{jl}^{ik}$ is summed to zero over any allele involved, so are $D_{j.}^{ik}$ and other disequilibrium measurements.

With Hardy-Weinberg and genotypic equilibria but linkage disequilibrium, model (7) leads to the following expression for the overall mean

$\begin{array}{c}E(G)=\mu +{\displaystyle \sum _{i,k}({\alpha }^i{\beta }^k)D_{\mathrm{..}}^{ik}}+{\displaystyle \sum _{j,l}({\alpha }_j{\beta }_l)D_{jl}^{\mathrm{..}}}\\ +{\displaystyle \sum _{i,j,k,l}({\delta }_j^i{\gamma }_l^k)D^{ik}{D}_{jl}}\end{array}$

where μ is the mean genotypic value under linkage equilibrium, and ${\Delta }_{\mu }={\displaystyle {\sum }_{i,k}({\alpha }^i{\beta }^k)}{D}_{\mathrm{..}}^{ik}+{\displaystyle {\sum }_{j,l}({\alpha }_j{\beta }_l)}{D}_{jl}^{\mathrm{..}}+{\displaystyle {\sum }_{i,j,k,l}({\delta }_j^i{\gamma }_l^k)}{D}_{\mathrm{..}}^{ik}{D}_{jl}^{\mathrm{..}}$ represents the departure from μ due to linkage disequilibrium and epistasis. If there is no epistasis, linkage disequilibrium does not affect the mean genotypic value. Similar results were given by [17]. Note that for marginal means of the genotypic values, we have

$\begin{array}{l}{G}_{\mathrm{..}}^{\mathrm{..}}={\displaystyle \sum _{i,j,k,l}{P}_{\mathrm{..}}^{ik}{P}_{jl}^{\mathrm{..}}{G}_{jl}^{ik},G_{\mathrm{..}}^{i.}=\frac{1}{p^i}{\displaystyle \sum _{j,k,l}{P}_{\mathrm{..}}^{ik}{P}_{jl}^{\mathrm{..}}{G}_{jl}^{ik}}},\\ G_{\mathrm{..}}^{ik}={\displaystyle \sum _{j,l}{P}_{jl}^{\mathrm{..}}{G}_{jl}^{ik},G_{j.}^{i.}=\frac{1}{p^i{p}_j}{\displaystyle \sum _{k,l}{P}_{jl}^{\mathrm{..}}{P}_{\mathrm{..}}^{ik}{G}_{jl}^{ik},}}\\ G_{j.}^{ik}=\frac{1}{p_j}{\displaystyle \sum _l{P}_{jl}^{\mathrm{..}}{G}_{jl}^{ik},\mathrm{...}}\end{array}$

and so on.

Then the question is what the genetic effects are in a disequilibrium population. Do Hardy-Weinberg and linkage disequilibria change the definition and values of genetic effects? The short answer to this question is "no" in a fully characterized model, but "yes" in a model that ignores some QTL or genetic effects. This is proved and discussed in [10]. With Hardy-Weinberg and linkage disequilibria, the genetic effects no longer correspond to the deviations from marginal means of genotypic values in a disequilibrium population. In a multiple regression model (7), the genetic effects are partial regression coefficients. These partial regression coefficients correspond to the simple regression coefficients, or deviations from marginal means of genotypic values, only in an equilibrium population. In a disequilibrium population, a direct analysis on the partial regression coefficients can be very complex (see the appendix of [10] for a relatively simple example). However, in a full model which includes all relevant loci and genetic effects, the model parameters depend only on how the regressors, i.e. x variables in (7), are defined and are independent of correlations between x variables, i.e. Hardy-Weinberg and linkage disequilibria. So, the genetic effects are still the same as those defined in the equilibrium population, although the population mean and marginal means of genotypic values are changed in a disequilibrium population.

Hardy-Weinberg and linkage disequilibria introduce correlation between x variables, thus covariances between different genetic effect components. Define

$\begin{array}{l}{A}_1={\displaystyle \sum _{i=1}^{n_1}{\alpha }^i{x}_{M_i}^{(1)}}+{\displaystyle \sum _{j=1}^{n_1}{\alpha }_j{x}_{F_j}^{(1)}}\\ D_1={\displaystyle \sum _{i,j}{\delta }_j^i{x}_{M_i}^{(1)}{x}_{F_j}^{(1)}}\\ A_2={\displaystyle \sum _{k=1}^{n_2}{\beta }^k{x}_{M_k}^{(2)}}+{\displaystyle \sum _{l=1}^{n_2}{\beta }_l{x}_{F_l}^{(2)}}\\ D_2={\displaystyle \sum _{k,l}{\gamma }_l^k{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}}\\ A_1{A}_2={\displaystyle \sum _{i,k}({\alpha }^i{\beta }^k)x_{M_i}^{(1)}{x}_{M_k}^{(2)}}+{\displaystyle \sum _{i,l}({\alpha }^i{\beta }_l)x_{M_i}^{(1)}{x}_{F_l}^{(2)}}\\ +{\displaystyle \sum _{j,k}(}{\alpha }_j{\beta }^k)x_{F_j}^{(1)}{x}_{M_k}^{(2)}+{\displaystyle \sum _{j,l}(}{\alpha }_j{\beta }_l)x_{F_j}^{(1)}{x}_{F_l}^{(2)}\\ A_1{D}_2={\displaystyle \sum _{i,k,l}({\alpha }^i{\gamma }_l^k})x_{M_i}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}\\ +{\displaystyle \sum _{j,k,l}({\alpha }_j{\gamma }_l^k})x_{F_j}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}\\ A_2{D}_1={\displaystyle \sum _{i,j,k}({\delta }_j^i{\beta }^k})x_{M_i}^{(1)}{x}_{F_j}^{(1)}{x}_{M_k}^{(2)}\\ +{\displaystyle \sum _{i,j,l}({\delta }_j^i{\beta }_l})x_{M_i}^{(1)}{x}_{F_j}^{(1)}{x}_{F_l}^{(2)}\\ D_1{D}_2={\displaystyle \sum _{i,j,k,l}({\delta }_j^i{\gamma }_l^k})x_{M_i}^{(1)}{x}_{F_j}^{(1)}{x}_{M_k}^{(2)}{x}_{F_l}^{(2)}\end{array}$

Then we can write

G = μ + A₁ + A₂ + D₁ + D₂ + A₁A₂ + A₁D₂ + A₂D₁ + D₁D₂

In a disequilibrium population, the partition of the genotypic variance becomes

$V_G={\displaystyle \sum _{i=1}^8{\displaystyle \sum _{j=1}^8{V}_{ij}}}=1^{T}V1$

where

V = (V _ij )_{8 × 8}

It is a symmetric matrix. In Appendix C, we give the detailed result for each component of the matrix with linkage disequilibrium, but assuming Hardy-Weinberg equilibrium.

For the rest of paper, when we discuss disequilibrium, we mainly discuss linkage disequilibrium and assume Hardy-Weinberg and genotypic equilibria which can be achieved by random mating in one generation. Hardy-Weinberg disequilibrium can be taken into account which will make results more complex and is thus omitted.

Reduced models

In many genetic applications, experimental population has some regular genetic structure by design. In these cases, the genetic model can be further simplified to reflect the experimental design structure. Also sometimes we may want to simplify the genetic model by imposing certain constrains or assumptions, such as the number of alleles, to increase the feasibility of analysis. In this section, we give a few reduced genetic models that are relevant to many genetic applications.

1. Backcross population or recombinant inbred population (haploid model)

Backcross population or recombinant inbred population is a common experimental design for QTL mapping study. By crossing two inbred lines, we can create a F₁ population. If we randomly backcross F₁ to one of the inbred lines, we have a backcross population. Let us assume that the cross is AA (paternal) × Aa (maternal). In a random-mating backcross population, there are only two possible genotypes at each segregating locus A _r A _r or A _r a _r , for r = 1, 2, ..., m, where m is the number of QTL. Since for the paternal gametes,

$z_{M_1}^{(r)}=\{\begin{array}{ll}1,\hfill & \text{for }{A}_r\text{ allele from paternal gamete}\hfill \\ 0,\hfill & \text{otherwise }\hfill \end{array}=1,$

and

$z_{M_2}^{(r)}=\{\begin{array}{ll}1,\hfill & \text{for }{a}_r\text{ allele from paternal gamete}\hfill \\ 0,\hfill & \text{otherwise }\hfill \end{array}=0,$

thus $x_{M_1}^{(r)}=z_{M_1}^{(r)}-1=0$ and $x_{M_2}^{(r)}=z_{M_2}^{(r)}=0$ for r = 1, 2, ..., m. For maternal gametes however,

$\begin{array}{c}{x}_{F_1}^{(r)}=\{\begin{array}{l}1/2,\text{ for }{A}_r\text{ from maternal gamete}\hfill \\ -1/2,\text{ otherwise}\hfill \end{array}\\ =-x_{F_2}^{(r)},\text{ for }r=1,2,\cdot \cdot \cdot ,m.\end{array}$

Thus the model becomes

$\begin{array}{c}G=\mu +{\displaystyle \sum _{r=1}^m{a}_r{x}_{F_1}^{(r)}+{\displaystyle \sum _{r<s}{b}_{rs}(x_{F_1}^{(r)}{x}_{F_1}^{(s)})}}\\ +{\displaystyle \sum _{r<s<t}{c}_{rst}(x_{F_1}^{(r)}{x}_{F_1}^{(s)}{x}_{F_1}^{(t)})+\cdots }\end{array}\left(8\right)$

where $a_r={\alpha }_1^{(r)}-{\alpha }_2^{(r)}$ is the substitution effect between homozygote genotype A _r A _r and heterozygote genotype $A_r{a}_r,b_{rs}=({\alpha }_1^{(r)}{\alpha }_1^{(s)})-({\alpha }_1^{(r)}{\alpha }_2^{(s)})-({\alpha }_2^{(r)}{\alpha }_1^{(s)})+({\alpha }_2^{(r)}{\alpha }_2^{(s)})$ is the interaction effect between loci r and $s,c_{rst}=({\alpha }_1^{(r)}{\alpha }_1^{(s)}{\alpha }_1^{(t)})-({\alpha }_1^{(r)}{\alpha }_1^{(s)}{\alpha }_2^{(t)})-({\alpha }_1^{(r)}{\alpha }_2^{(s)}{\alpha }_1^{(t)})+({\alpha }_1^{(r)}{\alpha }_2^{(s)}{\alpha }_2^{(t)})-({\alpha }_2^{(r)}{\alpha }_1^{(s)}{\alpha }_1^{(t)})+({\alpha }_2^{(r)}{\alpha }_1^{(s)}{\alpha }_2^{(t)})+({\alpha }_2^{(r)}{\alpha }_2^{(s)}{\alpha }_1^{(t)})-({\alpha }_2^{(r)}{\alpha }_2^{(s)}{\alpha }_2^{(t)})$, ..., and so on. Taking constraint conditions into account, we have α₁ = -α₂, β₁ = -β₂, and so on. Then, a _r = $2{\alpha }_1^{(r)}$, b _rs = 4(${\alpha }_1^{(r)}{\alpha }_1^{(s)}$), and c _rst = 8(${\alpha }_1^{(r)}{\alpha }_1^{(s)}{\alpha }_1^{(t)}$), and so on. With linkage equilibrium, the genetic effects as the partial regression coefficients of the model correspond to the simple regression coefficients. For example, for the substitution effect of locus r, a _r , it is the covariance between genotypic value G and substitution effect design variable $x_{F_1}^{(r)}$ divided by the variance of $x_{F_1}^{(r)}$. So in general, we have

$\begin{array}{c}{a}_r=E[(G-\mu )x_{F_1}^{(r)}]/E(x_{F_1}^{(r)2})\\ =E[G(z_{F_1}^{(r)}-1/2)]/(1/4)\\ =2[E(G|z_{F_1}^{(r)}=1)-E(G)]\\ b_{rs}=E[(G-\mu )x_{F_1}^{(r)}{x}_{F_1}^{(s)}]/[E(x_{F_1}^{(r)2})E(x_{F_1}^{(s)2})]\\ \begin{array}{l}=4[E(G|z_{F_1}^{(r)}=z_{F_1}^{(s)}=1)-E(G|z_{F_1}^{(r)}=1)\\ -E(G|z_{F_1}^{(s)}=1)+E(G)]\end{array}\\ c_{rst}=E[(G-\mu )x_{F_1}^{(r)}{x}_{F_1}^{(s)}{x}_{F_1}^{(t)}]/\\ [E(x_{F_1}^{(r)2})E(x_{F_1}^{(s)2})E(x_{F_1}^{(t)2})]\\ \begin{array}{l}=8[E(G|z_{F_1}^{(r)}=z_{F_1}^{(s)}=z_{F_1}^{(t)}=1)\\ -E(G|z_{F_1}^{(r)}=z_{F_1}^{(s)}=1)\\ -E(G|z_{F_1}^{(r)}=z_{F_1}^{(t)}=1)\\ -E(G|z_{F_1}^{(s)}=z_{F_1}^{(t)}=1)\\ +E(G|z_{F_1}^{(r)}=1)+E(G|z_{F_1}^{(s)}=1)\\ +E(G|z_{F_1}^{(t)}=1)-E(G)]\end{array}\end{array}$

The orthogonal partition of the genotypic variance in an equilibrium population is

$V_G=\frac{1}{4}{\displaystyle \sum _{r=1}^l{a}_r^2}+\frac{1}{4^2}{\displaystyle \sum _{r<s}{b}_{rs}^2}+\frac{1}{4^3}{\displaystyle \sum _{r<s<t}{c}_{rst}^2}+\cdots \left(9\right)$

As noted above, linkage disequilibrium does not change the values of genetic effects in a full model. The model parameters are still the same as those defined in the equilibrium population. However, in this case there is a simple relationship between the substitution effects at multiple loci and marginal means of genotypic values in a disequilibrium population [18]. This is noted here. Let $P_{rs}=P\{z_{F_1}^{(r)}=z_{F_1}^{(s)}=1\}$, and the digenic linkage disequilibrium be defined as

$D_{rs}=\text{Cov(}{z}_{F_1}^{(r)},z_{F_1}^{(s)})=E(x_{F_1}^{(r)}{x}_{F_1}^{(s)})$

Ignoring trigenic and higher order linkage disequilibria, we have

$E\left[(G-\mu )x_{F_1}^{(r)}\right]=\frac{1}{4}{a}_r+{\displaystyle \sum _{r\text{'}\ne r}{D}_{rr\text{'}}{a}_{r\text{'}}}$

$E[(G-\mu )x_{F_1}^{(r)}{x}_{F_1}^{(s)}]=\frac{1}{4^2}{b}_{rs}$

Therefore, the digenic interaction effects can be expressed as

$\begin{array}{l}{b}_{rs}=4^2[P_{rs}E(G|z_{F_1}^{(r)}=z_{F_1}^{(s)}=1)-D_{rs}\mu ]\\ -4[E(G|z_{F_1}^{(r)}=1)+E(G|z_{F_1}^{(s)}=1)-E(G)]\end{array}$

Then the substitution effects can be expressed as a function of marginal means in the disequilibrium population as

$\left(\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_l\end{array}\right)={(I+4D)}^{-1}(2q)$

where I is a m × m identity matrix, D = (D _ij )_{m × m}with all diagonal elements being zeros;

q = (q₁, q₂, ..., q _m ,)^T, with q _i = E(G|$z_{F_1}^{(i)}$ = 1) - E(G), for i = 1, 2, ..., m.

The partition of genetic variance with linkage disequilibrium is complex. Here we give details of the partition of genotypic variance for the following model

$G=\mu +{\displaystyle \sum _{r=1}^m{a}_r{x}_{F_1}^{(r)}}+{\displaystyle \sum _{r<s}{b}_{rs}}(x_{F_1}^{(r)}{x}_{F_1}^{(s)})\left(10\right)$

Let x _r = $x_{F_1}^{(r)}$ and x _s = $x_{F_1}^{(s)}$ to simplify the notation here. The genotypic variance is

$\begin{array}{l}{V}_G=V({\displaystyle \sum _r{a}_r}{x}_r)+2\text{Cov(}{\displaystyle \sum _r{a}_r{x}_r,{\displaystyle \sum _{r<s}{b}_{rs}{x}_r{x}_s)}}\\ +\text{V}({\displaystyle \sum _{r<s}{b}_{rs}{x}_r{x}_s)}\\ ={\displaystyle \sum _r{a}_r^2{p}_r(1-p_r)}+2{\displaystyle \sum _{r<s}{a}_r{a}_s{D}_{rs}}\\ +2{\displaystyle \sum _{r<s}[a_r{b}_{rs}(1-2p_r)D_{rs}+a_s{b}_{rs}(1-2p_s)D_{rs}]}\\ +2{\displaystyle \sum _{r<s<t}(a_r{b}_{st}+a_s{b}_{rt}+a_t{b}_{rs})D_{rst}}\\ +{\displaystyle \sum _{r<s}{b}_{rs}^2}[(1-2p_r)(1-2p_s)D_{rs}\\ +p_r(1-p_r)p_s(1-p_s)-D_{rs}^2]\\ +2{\displaystyle \sum _{r<s<t}\{b_{rs}{b}_{rt}[(1-2p_r)D_{rst}}\\ -p_r(1-p_r)D_{st}-D_{rs}{D}_{rt}]\\ +b_{rs}{b}_{st}[(1-2p_s)D_{rst}+p_s(1-p_s)D_{rt}-D_{rs}{D}_{st}]\\ +b_{rt}{b}_{st}[(1-2p_t)D_{rst}\\ +p_t(1-p_t)D_{rs}-D_{rt}{D}_{st}]\}\\ +2{\displaystyle \sum _{r<s<t<u}[b_{rt}{b}_{su}(D_{rstu}-D_{rt}}{D}_{su})\\ +b_{ru}{b}_{st}(D_{rstu}-D_{ru}{D}_{st})\\ +b_{rs}{b}_{tu}(D_{rstu}-D_{rs}{D}_{tu})]\left(11\right)\end{array}$

where

D _rst = E(x _r x _s x _t ) and D _rstu = E(x _r x _s x _t x _u )

are three locus and four locus linkage disequilibria. This is a general partition of genetic variance for a haploid model.

For the backcross population, it can be shown that D _rst = 0 (see Appendix D for both backcross and F₂ populations) and D _rstu = D _rs D _tu for loci r, s, t and u in this order under the assumption of no crossing-over interference. Also with this assumption, D _rt = 4D _rs D _st and D _rs = (1 - 2λ _rs )/4, where λ _rs is the recombination frequency between loci r and s. Since, p _r = p _s = 1/2, the variance becomes

$\begin{array}{ll}{V}_G=\hfill & \frac{1}{4}{\displaystyle \sum _r{a}_r^2+2{\displaystyle \sum _{r<s}{a}_r{a}_s{D}_{rs}+}}\hfill \\ \frac{1}{16}{\displaystyle \sum _{r<s}{b}_{rs}^2(1-16D_{rs}^2)}\hfill \\ +\frac{1}{2}{\displaystyle \sum _{r<s<t}[b_{rs}{b}_{rt}(1-16D_{rs}^2)D_{st}}\hfill \\ +b_{rt}{b}_{st}(1-16D_{st}^2)D_{rs}]\hfill \\ +2{\displaystyle \sum _{r<s<t<u}(b_{rt}{b}_{su}+b_{ru}{b}_{st})}\hfill \\ (1-16D_{st}^2)D_{rs}{D}_{tu}\hfill & \left(12\right)\hfill \end{array}$

In this partition of variance, the first summation term is the genetic variance due to the substitution effect of each QTL, the second summation term is the covariance between substitution effects of QTL pairs due to linkage disequilibrium, the third summation term is the genetic variance due to epistatic effects of QTL, and the fourth and fifth summation terms are the covariance between different epistatic effects of QTL due to linkage disequilibrium. There is no covariance between the main substitution effects and epistatic effects (see also [19]).

For a backcross population, the genetic interpretation of the substitution effect a _r depends on which parental line is backcrossed. In one backcross AA × Aa, the substitution effect is traditionally defined as the difference between the additive effect and dominance effects, and in the other backcross Aa × aa, it is the sum of the additive and dominance effects. Only with both backcrosses, can one estimate both additive and dominance effects separately (for example [20]).

The same model also applies to a recombinant inbred population which is another very popular experimental design for QTL mapping study. For a recombinant inbred population, the substitution effects of QTL are the additive effects and the epistatic effects are the additive × additive interaction effects. Statistical methods to map QTL and to estimate various components of the genetic variance due to QTL including epistasis has been developed through the maximum likelihood approach [19, 21]. In a few cases where the method was applied, we estimated, for the first time, how the quantitative genetic variance was partitioned into various components in designed experimental populations. For example, Weber et al. [22] reported the result of QTL mapping for wing shape on the third chromosome of Drosophila melanogaster from a cross of divergent selection lines. From 519 recombinant inbred lines, 11 QTL were mapped on the third chromosome. Nine QTL pairs showed significant epistatic effects. The total genetic variance amounts to 95.5% of the phenotypic variance in the recombinant inbred lines with phenotypes measured and averaged over 50 male flies for each recombinant inbred line. The partition of the genetic variance is as follows (see Table 6 and 7 of [22]): 27.4% due to the variances of additive effects (equivalent to the first summation term of (12)); 67.3% due to the covariances between additive effects (the second summation term); 7.2% due to the variances of epistatic effects (the third summation term); and -6.0% due to the covariances between epistatic effects (the fourth and fifth summation terms). The covariances between additive and epistatic effects, expected to be 0, account for -0.4% due to sampling. Similar kind of partition of the genetic variance is also observed in a group of 701 second chromosome recombinant inbred lines from a cross of the same divergent selection lines (see Table 4 and 5 of [23]). See also [20] for another example.

2. F₂ population

F₂ is created from a cross between pairs of F₁ individuals. It is also a very popular experimental design for QTL mapping study. The advantage of this design is that both additive and dominance effects of a QTL can be estimated as well as various epistatic effects. The design also has more statistical power for QTL detection as compared to a backcross population. In a random-mating F₂ population, there are only two alleles at each segregating locus and allelic frequencies are expected to be one half if there is no segregation distortion.

Let us consider only two loci first. Let A and a denote the two alleles at locus 1, and B and b at locus 2. In this case, $x_{M_1}^{(1)}=-x_{M_2}^{(1)}$ and $x_{F_1}^{(1)}=-x_{F_2}^{(1)}$. Assuming $G_{jl}^{ij}=G_{ik}^{jl}=G_{jk}^{il}=G_{il}^{jk}$, it also holds that α¹ = α₁ = -α² = -α₂, ${\delta }_1^1={\delta }_2^2=-{\delta }_1^2$, and so on. The additive term for locus 1 then becomes

$A_1={\alpha }^1(x_{M_1}^{(1)}-x_{M_2}^{(1)})+{\alpha }_1(x_{F_1}^{(1)}-x_{F_2}^{(1)})=2{\alpha }_1{w}_1$

with

$w_1=x_{M_1}^{(1)}+x_{F_1}^{(1)}=\{\begin{array}{l}1,\text{ for }AA\text{ at locus 1}\hfill \\ \text{0, for }Aa\text{ at locus 1}\hfill \\ -1,\text{ for }aa\text{ at locus 1}\hfill \end{array}\left(13\right)$

and the dominance term is

$D_1=2({\delta }_1^1)(x_{M_1}^{(1)}{x}_{F_1}^{(1)}+x_{M_1}^{(1)}{x}_{F_1}^{(1)})=(-2)({\delta }_1^1)v_1$

with

$v_1=(-2)x_{M_1}^{(1)}{x}_{F_1}^{(1)}=\{\begin{array}{c}-1/2,\text{ for }AA\text{ at locus 1}\\ 1/2,\text{ for }Aa\text{ at locus 1}\\ -1/2,\text{ for }aa\text{ at locus 1}\end{array}\left(14\right)$

Note that the v variable in this section for F₂ differ, by a factor -2, from the v variable in the next section for a general two-allele model to conform to the usual definition for the F₂ model. Similarly, for locus 2

A₂ = 2β₁w₂ and D₂ = (-2)(${\gamma }_{\text{1}}^{\text{2}}$)v₂

with

$w_2=\{\begin{array}{c}1,\text{ for }BB\text{ at locus 2}\\ 0,\text{ for }Bb\text{ at locus 2}\\ -1,\text{ for }bb\text{ at locus 2}\end{array}$

and

$v_2=\{\begin{array}{c}-1/2,\text{ for }BB\text{ at locus 2}\\ 1/2,\text{ for }Bb\text{ at locus 2}\\ -1/2,\text{ for }bb\text{ at locus 2}\end{array}\left(15\right)$

The model can then be written as

$\begin{array}{c}G=\mu +a_1{w}_1+d_1{v}_1+a_2{w}_2+d_2{v}_2\\ +{(aa)}_{12}(w_1{w}_2)+{(ad)}_{12}(w_1{v}_2)\\ +{(da)}_{12}(v_1{w}_2)+{(dd)}_{12}(v_1{v}_2)\end{array}\left(16\right)$

where the parameters are related as a₁ = 2α₁, a₂ = 2β₂, d₁ = - $-2{\delta }_1^1$, d₂ = -$-2{\gamma }_1^1$, (aa)₁₂ = 4(α₁β₁), (ad)₁₂ = 8(${\alpha }_1{\gamma }_1^1$), (da)₁₂ = 8(${\delta }_1^1{\beta }_1$), (dd)₁₂ = 16(${\delta }_1^1{\gamma }_1^1$). With random mating and linkage equilibrium, we have

$\begin{array}{c}{a}_1=E[G-\mu )w_1]/E(w_1^2)=2(G_{1.}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}})\\ d_1=E[(G-\mu )v_1]/E(v_2^1)\\ =(-2)(G_{1.}^{1.}-2G_{1.}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}})\\ a_2=2(G_{.1}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}})\\ d_2=(-2)(G_{.1}^{.1}-2G_{.1}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}})\\ (aa{)}_{12}=E[(G-\mu )w_1{w}_2]/E(w_1^2{w}_2^2)\\ =4(G_{\mathrm{..}}^{11}-G_{1.}^{\mathrm{..}}-G_{.1}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}}\\ (ad{)}_{12}=E[(G-\mu )w_1{v}_2]/E(w_1^2{v}_2^2)\\ =(-4)(G_{11}^{.1}-2G_{11}^{\mathrm{..}}-G_{.1}^{.1}+G_{1.}^{\mathrm{..}}+2G_{.1}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}})\\ (da{)}_{12}=E[(G-\mu )v_1{w}_2]/E(v_1^2{w}_2^2)\\ =(-4)(G_{11}^{1.}-2G_{11}^{\mathrm{..}}-G_{1.}^{1.}+G_{.1}^{\mathrm{..}}+2G_{1.}^{\mathrm{..}}-G_{\mathrm{..}}^{\mathrm{..}})\\ (dd{)}_{12}=E[(G-\mu )v_1{v}_2]/E(v_1^2{v}_2^2)\\ =4(G_{11}^{11}-2G_{11}^{1.}-2G_{11}^{.1}+G_{1.}^{1.}+G_{.1}^{.1}+4G_{11}^{\mathrm{..}}\\ -2G_{1.}^{\mathrm{..}}-2G_{.1}^{\mathrm{..}}+G_{\mathrm{..}}^{\mathrm{..}})\end{array}$

The orthogonal partition of the genotypic variance is

$\begin{array}{c}{V}_G=\frac{1}{2}{a}_1^2+\frac{1}{4}{d}_1^2+\frac{1}{2}{a}_2^2+\frac{1}{4}{d}_2^2+\frac{1}{4}{(aa)}_{12}^2\\ +\frac{1}{8}{(ad)}_{12}^2+\frac{1}{8}{(da)}_{12}^2+\frac{1}{16}{(dd)}_{12}^2.& \left(17\right)\end{array}$

Recently, Kao and Zeng [18] have examined many genetic and statistical issues of the above F₂ model and the effects of linkage disequilibrium. As we have shown here, the F₂ model is a special case of Cockerham model with two alleles at each locus and all allelic frequencies being 1/2.

Now we give the partition of genetic variance for m loci with epistasis and linkage disequilibrium in the F₂ population. Generalizing model (16) to m loci and ignoring the trigenic and higher order epistasis, we have the following model

$\begin{array}{c}G=\mu +{\displaystyle \sum _{r=1}^m{a}_r{w}_r+{\displaystyle \sum _{r=1}^m{d}_r{v}_r+{\displaystyle \sum _{r<s}{(aa)}_{rs}(w_r{w}_s)}}}\end{array}$