Finite Mixture Models (FMM) are widely used in the analysis of biological, economic and sociological data. For a comprehensive survey of different statistical techniques based on FMMs, see [9]. Mixtures with Varying Concentrations (MVC) is a subclass of these models in which the mixing probabilities are not constant, but vary for different observations (see [4, 5]).

In this paper we consider application of the jackknife technique to the estimation of asymptotic covariance matrix (the covariance matrix for asymptotically normal estimator, ACM) in the case when the data are described by the MVC model. The jackknife is a well-known resampling technique usually applied to i.i.d. samples (see Section 5.5.2 in [11], Chapter 4 in [15], Chapter 2 in [12]). On the jackknife estimates of variance for censored and dependent data, see [14]. Its modification to the case of the MVC model in which the observations are still independent but not identically distributed needs some efforts.

We obtained a general theorem on consistency of the jackknife estimators for ACM for moment estimators in the MVC models and apply this result to construct confidence sets for regression coefficients in linear errors-in-variables models for MVC data. On general errors-in-variables models, see [2, 3, 8]. The model and the estimators for the regression coefficients considered in this paper was proposed in [6], where the asymptotic normality of these estimates is shown.

The rest of the paper is organized as follows. In Section 2 we introduce the MVC model and describe the estimation technique for these models based on weighted moments. In Section 3 the jackknife estimates for the ACM are introduced and conditions of their consistency formulated. Section 4 is devoted to the algorithm of fast computation of the jackknife estimates. In Section 5 we apply the previous results to construct confidence sets for linear regression coefficients in errors-in-variables models with MVC. In Section 6 results of simulations are presented. In Section 7 we present results of application of the proposed technique to analyze sociological data. Proofs are placed in Section 8. Section 9 contains concluding remarks.

We consider a dataset in which each observed subject O belongs to one of M subpopulations (mixture components). The number κ(O) of the population which O belongs to is unknown. We observe d numeric characteristics of O which form the vector ξ(O)=(ξ1(O),…,ξd(O))T∈Rd of observable variables. The distribution of ξ(O) may depend on the component κ(O) : Fξ(m)(A)=P{ξ(O)∈A|κ(O)=m},m=1,…,M, where A is any Borel subset of Rd .

We observe variables of n independent subjects ξj=ξ(Oj) . The probability to obtain j-th subject from m-th component pj(m)=P{κ(Oj)=m} can be considered as the concentration of the m-th component in the mixture when the j-th observation was made. The concentrations are known and can vary for different observations.

So, the distribution of ξj is described by the model of mixture with varying concentrations: (1) P{ξj∈A}= ∑m=1Mpj(m)Fξ(m)(A) We will denote by μ(m)=E(m)[ξ]=E[ξ(O)|κ(O)=m]=∫RdxFξ(m)(dx) the vector of theoretical first moments of the m-th component distribution. In what follows, Cov(m)[ξ] means the covariance of ξ(O) for the m-th component, Var(m)[ξl] means the variance of ξl(O) for this component and so on.

To estimate μ(k) by observations ξ1 , …, ξn one can use the weighted sample mean (2) ξ¯;n(k)=ξ¯(k)= ∑j=1naj(k)ξj, where aj(k)=aj;n(k) are some weights dependent on components’ concentrations, but not on the observed ξj=ξj;n . (In what follows we denote by the subscript ;n that the corresponding quantity is considered for the sample size n. In most cases this subscript is dropped to simplify notations.)

To obtain unbiased estimates in (2) one needs to select the weights satisfying the assumption (3) ∑j=1naj(k)pj(m)=1ifk=m,0ifk≠m. Let us denote Ξ=(ξ1T,…,ξn)T=ξ11…ξ1d⋮⋱⋮ξn1…ξnd,a=a1(1)…a1(M)⋮⋱⋮an(1)…an(M)andp=p1(1)…p1(M)⋮⋱⋮pn(1)…pn(M). Then p․(m)=(p1(m),…,pn(m))T , pj․=(pj(1),…,pj(M))T and the same notation is used for the matrix a.

In this notation the unbiasedness condition (3) reads (4) aTp=E, where E means the M×M unit matrix.

There can be many choices of a satisfying (4). In [4, 5] minimax weights are considered defined by2²

In fact, in [4] and [5] the weights are defined as a˜=na and ξ¯(k)=1n∑j=1na˜j(k)ξj . In this paper we adopt notation which allows to simplify formulas for fast estimator calculation in Section 4.

To describe the asymptotic behavior of ξ¯;n(k) as n→∞ , we will calculate its covariance matrix.

Notice that Cov[ξj]=E[ξjξjT]−E[ξj]E[ξj]T= ∑m=1Mpj(m)Σ(m)− ∑m,l=1Mpj(m)pj(l)μ(m)(μ(l))T, where Σ(m)=Cov(m)[ξ]=Cov[ξ(O)|κ(O)=m] . So, nCovξ¯(k)= ∑m=1M⟨(a(k))2p(m)⟩;nΣ(m)− ∑m,l=1M⟨(a(k))2p(m)p(l)⟩;nμ(m)(μ(l))T, where ⟨(a(k))2p(m)⟩;n=n ∑j=1n(aj(k))2pj(m),⟨(a(k))2p(m)p(l)⟩;n=n ∑j=1n(aj(k))2pj(m)pj(l). Assume that the limits (6) ⟨(a(k))2p(m)p(l)⟩∞=deflimn→∞⟨(a(k))2p(m)p(l)⟩;n exist. Then the limits ⟨(a(k))2p(m)⟩∞=deflimn→∞⟨(a(k))2p(m)⟩;n exist also, since due to (3) we have ∑l=1Mpjl=1 for all j.

So, under this assumption, (7) nCov[ξ¯(k)]→Σ∞asn→∞, where Σ∞=def ∑m=1M⟨(a(k))2p(m)⟩∞Σ(m)− ∑m,l=1M⟨(a(k))2p(m)p(l)⟩∞μ(m)(μ(l))T.

This theorem is a simple corollary of Theorem 4.3 in [5].

In what follows, we will consider unknown parameters of the component distribution Fξ(k) , which can be represented in the form (8) ϑ=ϑ(k)=H(μ(k)), where H:Rd→Rq is some known function. A natural estimator for such parameter by the sample ξ1 , …, ξn is (9) ϑˆ=ϑˆ;n(k)=H(ξ¯;n(k)). Then asymptotic behavior of this estimator is described by the following theorem.

This theorem is a simple implication from our Theorem 1 and Theorem 3 in Section 5, Chapter 1 of [1].

So, V∞ defined by (10) is the ACM of the estimator ϑˆ(k) (the covariance matrix of the limit normal distribution of the normalized difference between the estimator and the estimated parameter). If it was known one could use it to construct tests for hypotheses on ϑ(k) or to derive confidence set for ϑ(k) . In fact, for most estimators the ACM is unknown. Usually some estimate of V∞ is used to replace its true value in statistical algorithms.

The jackknife is one of most general techniques of ACM estimation. Let ϑˆ be any estimator of ϑ by the data ξ1 , …, ξn : ϑˆ=ϑˆ(ξ1,…,ξn). Consider estimates of the same form which are calculated by all observations without one ϑˆi−=ϑˆ(ξ1,…,ξi−1,ξi+1,…,ξn). Then the jackknife estimator for V∞ is defined by (11) Vˆ;n=Vˆ;n(k)=n ∑i=1n(ϑˆi−−ϑˆ)(ϑˆi−−ϑˆ)T. In our case ϑˆ=H(ξ¯(k)) , so (12) ϑˆi−=H(ξ¯i−(k)), where (13) ξ¯j−(k)=∑j≠iaji−(k)ξj. Here ai−=(aji−(m),j=1,…,n,m=1,…,M) is the minimax weights matrix calculated by the matrix pi− of concentrations of all observations except i-th one. That is, pi−=(p1․,…,pi−1․,0,pi+1․,…,pn․)T , (14) ai−=pi−Γi−−1, (15) Γi−=pi−Tpi−. Notice that 0 is placed at the i-th row of pi− as a placeholder only, to preserve the numbering of the rows in pi− and ai− , which corresponds to the numbering of subjects in the sample.

For proof see Section 8.

Direct calculation of Vˆ;n by (11)–(15) needs ∼Cn2 elementary operations. Here we consider an algorithm which reduces the computational complexity to ∼Cn operations (linear complexity).

Notice that Γi−=Γ−pi․(pi․)T . So (16) (Γi−)−1=Γ−1+11−hiΓ−1pi․(pi․)TΓ−1, where (17) hi=(pi․)TΓ−1pi․. (Formula (16) can be demonstrated directly by checking Γi−−1Γi−=E . It is also a corollary to the Serman–Morrison–Woodbury formula, see A.9.4 in [13].)

Let us denote ξ¯i−=ξ¯i−1(1)…ξ¯i−d(1)⋮⋱⋮ξ¯i−1(M)…ξ¯i−d(M)=((ξ¯i−(1))T,…,(ξ¯i−(M))T)T and (18) ξ¯=((ξ¯(1))T,…,(ξ¯(M))T)T. Then ξ¯i−=(Γi−)−1pi−Ξi− , where Ξi−=(ξ1,…,ξi−1,0,ξi+1,…,ξn)T . (Zero at the i-th row is a placeholder as in the matrix pi− .) Applying (16) one obtains ξ¯i−=Γ−1(pi−)TΞi−+11−hiΓ−1pi․(pi․)TΓ−1(pi․)TΞi−. This together with (pi−)TΞi−=pTΞ−pi․ξiT implies (19) ξ¯i−=ξ¯+11−hiai․((pi․)Tξ¯−ξiT). Then the following algorithm allows one to calculate Vˆ(m) for all m=1,…,M at once by ∼Cn operations.

In this section we consider a mixture of simple linear regressions with errors in variables. A modification of orthogonal regression estimation technique was proposed for the regression coefficients estimation in [6]. We will show how the jackknife ACM estimators from Section 3 can be applied in this case to construct confidence sets for the regression coefficients.

Recall the errors-in-variables regression model in the context of mixture with varying concentrations.

We consider the case when each subject O has two variables of interest: x(O) and y(O) . These variables are related by a strict linear dependence with coefficients depending on the component that O belongs to: (20) y(O)=b0(κ(O))+b1(κ(O))x(O), where b0(m) , b1(m) are the regression coefficients for the m-th component.

The true values of x(O) and y(O) are unobservable. These variables are observed with measurement errors (21) X(O)=x(O)+εX(O),Y(O)=y(O)+εY(O). Here we assume that the errors εX(O) and εY(O) are conditionally independent given κ(O)=m , (22) E(m)εX=E(m)εY=0andVar(m)εX=Var(m)εY=σ(m)2 for all m=1,…,M . So the distributions of εX(O) and εY(O) can be different, but their variances are the same for a given subject. We assume that σ(m)2>0 0$]]>, m=1,…,M , and are unknown.

As in Section 2 we observe a sample (X(Oj),Y(Oj))T=(Xj,Yj)T , j=1,…,n , from the mixture with known concentrations pj(m)=P{κ(Oj)=m} .

In the case of homogeneous sample, when there is no mixture, the classical way to estimate b0 and b1 is orthogonal regression. That is, the estimator is taken as the minimizer of the total least squares functional which is the sum of squares of minimal Euclidean distances from the observation points to the regression line. The modification of this technique for mixtures with varying concentrations proposed in [6] leads to the following estimators for b0(k) and b1(k) : (23) bˆ1(k)=SˆXX(k)−SˆYY(k)+(SˆXX(k)−SˆYY(k))2+4(SˆXY(k))22SˆXY(k),bˆ0(k)=Y¯k−bˆ1(k)X¯(k), where X¯(k)= ∑j=1naj(k)Xj,Y¯(k)= ∑j=1naj(k)Yj,SˆXX(k)= ∑j=1naj(k)(Xj−X¯(k))2,SˆYY(k)= ∑j=1naj(k)(Yj−Y¯(k))2,SˆXY(k)= ∑j=1naj(k)(Xj−X¯(k))(Yj−Y¯(k)). Conditions of consistency and asymptotic normality of these estimators are given in Theorems 5.1 and 5.2 from [6]. For example, under the assumptions of Theorem 4 we obtain n(ϑˆ;n(k)−ϑ(k))→N(0,V∞(k)), where ϑ(k)=(b0(k),b1(k))T .

The ACM V∞ of the estimator is given by formula (21) in [6]. This formula is rather complicated and involves theoretical moments of unobservable variables x(O) , εX(O) and εY(O) . So it is natural to estimate V∞ by the jackknife technique, which doesn’t need to know or estimate these moments.

Notice that the estimator (bˆ0(k),bˆ1(k))T can be represented in terms of Section 3 if we expand the space of observable variables including quadratic terms. That is, we consider the sample ξj=(Xj,Yj,(Xj)2,(Yj)2,(XjYj)2)T,j=1,…,n. Then the estimator ϑˆ;n=(ϑˆ;n1,ϑˆ;n2)T=(bˆ0(k),bˆ1(k))T defined by (23) can be represented in form (9) with twice continuously differentiable function H if Var(k)[x]≠0 and b1(k)≠0 .

So we can apply the technique developed in Sections 3–4. Let us define the estimator Vˆ;n(k) for V∞(k) by (11).

This theorem is a simple combination of Theorem 3 and Theorem 5.2 from [6].

In what follows we assume that V∞(k) is nonsingular. This assumption holds, e.g. if for all m the distributions of x(O) , εX(O) and εY(O) given κ(O)=m are absolutely continuous with continuous PDFs. (The proof of this fact is rather technical, so we do not present it here.)

We can construct a confidence set (ellipsoid) for the unknown parameter ϑ(k) applying the Theorem 4 by the usual way. Namely, let for any t∈R2 T;n(t)=(t−ϑˆ;n(k))T(Vˆ;n(k))−1(t−ϑˆ;n(k)). Then in the assumptions of Theorem 4, if detV∞(k)≠0 , (24) T;n(ϑ(k))⟶Wη,asn→∞, where η is a random variable (r.v.) with chi-square distribution with 2 degrees of freedom.

Consider Bα;n={t∈R2:T;n(t)≤Qη(1−α)} , where Qη(α) means the quantile of level α for the r.v. η. By (24) (25) P{ϑ(k)∈Bα;n}→1−α, so Bα;n is an asymptotic confidence set for ϑ(k) of level α.

To assess performance of the proposed technique we performed a small simulation study. In the following three experiments we calculated covering frequencies of confidence sets for regression coefficients in the model (20)–(22) constructed by (25) and corresponding one-dimensional confidence intervals.

In all experiments for sample size n=100 through 5000 we generated B=1000 samples and calculated estimates for the parameters and corresponding confidence sets. The one-dimensional confidence intervals for bi(k) were calculated by the standard formula bˆi;n(k)−λα/2vˆii;n(k)n,bˆi;n(k)+λα/2vˆii;n(k)n, where vˆii;n(k) is the i-th diagonal entry of the matrix Vˆ;n(k) , λα/2 is the quantile of level 1−α/2 for the standard normal distribution. The confidence level for the sets and intervals was taken α=0.05 .

Then the numbers of cases when the confidence set covers the true value of the estimated parameter were calculated and divided by B. These are the covering frequencies reported in the tables below.

In all the experiments we considered two-component mixture ( M=2 ) with the concentrations of components pj;n1=j/n,pj;n2=1−j/n. The regression coefficients were taken as b0(1)=1/2,b1(1)=2,b0(2)=−1/2,b1(2)=−1/3, and the distribution of the true (unobservable) regressor x(O) was N(0,2) for κ(O)=1 and N(1,2) for κ(O)=2 . Experiment 1.

In this experiment we let εX and εY∼N(0,0.25) . The variance of the errors is so small that the regression coefficients can be estimated with no difficulties even for small sample sizes.

The covering frequencies for confidence sets are presented in Table 1. It seems that they approach the nominal covering probability 0.95 with satisfactory accuracy for sample sizes n≥1000 .

We would like to demonstrate advantages of the proposed technique by application to the analysis of the External Independent Testing (EIT) data (see [7]). EIT is a set of exams for high school graduates in Ukraine which must be passed for admission to universities. We use data on EIT-2016 from the official site of Ukrainian Center for Educational Quality Assessment.3³

https://zno.testportal.com.ua/stat/2016

In this presentation we consider only the data on scores on two subjects: Ukrainian language and literature (Ukr) and on Mathematics (Math). The scores range from 100 to 200 points. (We have excluded the data on persons who failed on one of the exams or didn’t pass these exams at all.) EIT-2016 contain such data on 246 thousands of examinees. The information on the region (Oblast) of Ukraine in which each examinee attended the high school is also available in EIT-2016.

Our aim is to investigate how dependence between Ukr and Math scores differs for examinees grown up in different environments There can be, e.g. an environment of adherents of Ukrainian culture and Ukrainian state, or in the environment of persons critical toward the Ukrainan independence. EIT-2016 doesn’t contain information on such issues. So we use data on Ukrainian Parliament (Verhovna Rada) election results to deduce approximate proportions of adherents of different political choices in different regions of Ukraine.

We divided adherents of 29 parties and blocks that took part in the elections into three large groups, which are the components of our mixture:

Combining these data with EIT-2016 we obtain the sample (Xj,Yj) , j=1,…,n , where Xj is the Math score of the j-th examinee, Yj is his/her Ukr score. The concentrations of components (pj1,pj2,pj3) are taken as frequencies of adherents of corresponding political choice at the region where the j-th examinee attended the high school.

In [7] the authors propose to use classical linear regression model (in which the error appears in the response only) to describe dependence between Xj and Yj in these data. But the errors-in-variables model can be more adequate since the causes which deteriorate ideal functional dependence Ukr=b0+b1 Math can affect both Math and Ukr scores causing random deviations of, maybe, the same dispersion for each variable.

So, in this presentation, we assumed that the data are described by the model (20)–(22), where κ(O)=1,2,3 means the component (environment at which the person O was grown up) corresponding to one of three political choices given above. (Lower and upper endpoints of confidence intervals are given in columns named low and upp correspondingly.)

In this model we calculated the confidence intervals of the level α=0.05/3≈0.0167 to derive the unilateral level α=0.05 in comparisons of three intervals derived for three different components. The results are presented in Table 4. We observe that the obtained intervals are rather narrow. They don’t intersect for different components. So, the regression coefficients for different components are significantly different. (Of course, it is so only if our theoretical model of the data distribution is adequate.)

The orthogonal regression lines corresponding to different components are presented on Fig 1. The solid line corresponds to the Pro-component, the dashed line for the Contra-component and the dotted line for the Neutrals.

These results have simple and plausible explanation. Say, in the Pro-component the success in Ukr positively correlates with the general school successes, so with Math scores, too. It is natural for persons who are interested in Ukrainian culture and literature. In the Contra-component the correlation is negative. Why? The persons with high Math grades in this component do not feel the need to learn Ukrainian. But the persons with less success in Math try to improve their average score (by which the admission to universities is made) by increasing their Ukr score. The Neutral component shows positive correlation between Math and Ukr, but it is less then the correlation in the Pro-component.

Surely, these explanations are too simple to be absolutely correct. We consider them only as examples of hypotheses which can be deduced from the data by the proposed technique.

To demonstrate Theorem 3 we need three lemmas. Below the symbols C and c mean finite constants, maybe different. Lemma 1.

Assume that detΓ∞>0 0$]]>. Then: 1.

supj=1,…,n;m=1,…,M|aj;nm|=O(n−1) .

Let ηj=±naj;n(k)ξjl . Then Bn=n2∑j=1n(aj;n(k))2Varξjl∼Cn by Lemma 1. Assumption 2 implies that the assumption of Proposition 1 holds. So, P ∑j=1nnaj;n(k)ξjl>b2CnloglogCn→0. b\sqrt{2Cn\log \log Cn}\right\}\to 0.\]]]> This implies the statement of the lemma.  □