VMSTA Modern Stochastics: Theory and Applications 2351-6054 2351-6046 2351-6046 VTeXMokslininkų g. 2A, 08412 Vilnius, Lithuania VMSTA154 10.15559/20-VMSTA154 Research Article Prediction in polynomial errors-in-variables models KukushAlexanderalexander_kukush@univ.keiv.ua∗ SenkoIvanivan_senko@ukr.net Taras Shevchenko National University of Kyiv, Kyiv, Ukraine Corresponding author. 2020 255202072203219 2442020 452020 652020 © 2020 The Author(s). Published by VTeX2020 Open access article under the CC BY license.

A multivariate errors-in-variables (EIV) model with an intercept term, and a polynomial EIV model are considered. Focus is made on a structural homoskedastic case, where vectors of covariates are i.i.d. and measurement errors are i.i.d. as well. The covariates contaminated with errors are normally distributed and the corresponding classical errors are also assumed normal. In both models, it is shown that (inconsistent) ordinary least squares estimators of regression parameters yield an a.s. approximation to the best prediction of response given the values of observable covariates. Thus, not only in the linear EIV, but in the polynomial EIV models as well, consistent estimators of regression parameters are useless in the prediction problem, provided the size and covariance structure of observation errors for the predicted subject do not differ from those in the data used for the model fitting.

 Prediction multivariate errors-in-variables model polynomial errors-in-variables model ordinary least squares consistent estimator of best prediction confidence interval 62J05 62J02 62H12 Introduction

We deal with errors-in-variables (EIV) models which are widely used in system identification [10], epidemiology [2], econometrics [12], etc. In such regression models (with unknown parameter β), the response variable y depends on the covariates z and ξ, where z is observed precisely and ξ is observed with error. We consider the classical measurement error δ, i.e., instead of ξ the surrogate data x=ξ+δ is observed; moreover, the model is structural, i.e. z, ξ and δ are mutually independent, and we have i.i.d. copies of the model ( zi , ξi , δi , xi=ξi+δi , yi ), i=1,…,n . The measurement error can be nondifferential, when the distribution of y given ξ,z,x depends only on ξ,z , and differential, otherwise [2, Section 2.5].

The present paper is devoted to the prediction of the response variable from ξ and z. Based on the observations ( yi , zi , xi ), i=1,…,n , and given new values z0 and x0 of z and x variables, we want to predict either the new y0 (this procedure is called individual prediction) or the exact relation η0=Ey0z0,ξ0 , where ξ0 is a new value for ξ (this procedure is called mean prediction). Both prediction problems are important in econometrics [5]. The individual prediction is used in the Leave-one-out cross-validation procedure.

The best mean squared error individual predictor is yˆ0=Ey0z0,x0 and the best mean squared error predictor of  η0 is ηˆ0=Eη0z0,x0. For the nondifferential measurement error, ηˆ0=EEy0z0,ξ0,x0z0,x0]=Ey0z0,x0=yˆ0, and the best mean predictor coincides with the best individual predictor, but this needs not to hold for the differential measurement error.

Both predictors (1) and (2) are unfeasible, because they involve unknown model parameters. Our goal is to construct consistent estimators of the predictors as the sample size n grows.

The nonparametric individual prediction under errors in covariates is studied in [7]. Below we consider only parametric models.

For scalar linear EIV models with normally distributed ξ and δ, it is stated in [4, Section 2.5.1] that the ordinary least squares (OLS) predictor should be used even when dealing with the EIV model. This is quite surprising, since the OLS estimator of β is inconsistent due to the attenuation effect [4]. In fact, there is no surprise that in a Gaussian model the linear OLS estimator provides a consistent prediction, since the Gaussian dependence is always linear. In the present paper, we consider a non-Gaussian regression model, since the distribution of the observable covariate z is not assumed Gaussian; therefore, the consistency of OLS predictions in such a model is a nontrivial feature.

We confirm the assertion, that the OLS estimator yields a suitable prediction under the model validity, for two kinds of EIV models: multivariate linear and polynomial. For this purpose, we just follow the recommendation of [4, Section 2.6] and analyze the regression of y on the observable z and x. In other nonlinear EIV models, the OLS predictor (contaminated from the initial regression y on z,ξ , where we naively substitute x for ξ) is inconsistent; instead the least-squares predictor can be used from the regression y on z,x .

The paper is organized as follows. In Sections 2 and 3, we state the results on prediction in multivariate linear and polynomial EIV models, respectively. Section 4 studies briefly some other nonlinear EIV models, and Section 5 concludes.

Through the paper, all vectors are column ones, E stands for the expectation and acts as an operator on the total product, and Covx denotes the covariance matrix of a random vector x. By  Ip we denote the identity matrix of size p. For symmetric matrices A and B of the same size, A>B B$]]> and A≥B means that A−B is positive definite or positive semidefinite, respectively.

 Prediction in a multivariate linear EIV model Model and main assumptions

Consider a multivariate linear EIV model with the intercept term (structural case): y=b+CTz+BTξ+e+ϵ, x=ξ+δ. Here the random vector y is the response variable distributed in  Rd ; the random vector z is the observable covariate distributed in  Rq , the random vector ξ is the unobservable (latent) covariate distributed in  Rm ; x is the surrogate data observed instead of ξ; e+ϵ is the random error in y, δ is the measurement error in the latent covariate; C∈Rq×d , B∈Rm×d and b∈Rd contain unknown regression parameters, where b is the intercept term. The random vector e models the error in the regression equation, and ϵ models the measurement error in y; ϵ can be correlated with δ.

Such models are studied, e.g., in [11, 10, 9] in relation to system identification problems and numerical linear algebra. We list the model assumptions.

Three vectors z, ξ, e and the augmented measurement error vector ϵT,δTT are independent with finite 2nd moments; the errors ϵ and δ can be correlated.

The covariance matrices Σz:=Cov(z) and Σx:=Cov(x) are nonsingular.

The errors e, ϵ and δ have zero means.

The errors ϵ, δ and covariate ξ are jointly Gaussian.

Introduce the cross-covariance matrix Σϵδ:=EϵδT. The classical measurement error δ is nondifferential if, and only if, ϵ and δ are independent, i.e. Σϵδ=0 (see Section 1 for the definition of the nondifferential error).

We denote also μ=Ex,Σξ=Cov(ξ),Σe=Cov(e),Σϵ=Cov(ϵ),Σδ=Cov(δ),Σ11=block‐diag(Σξ,Σϵ),Σ12=ΣξΣϵδ,Σ22=Σx. Thus, Σ11 is a block-diagonal matrix, and sometimes we will use Σ22 for the covariance matrix of x.

 Regression of <italic>y</italic> on <italic>z</italic> and <italic>x</italic>

Assume conditions (i) to (iv).

The response variable (3) can be represented as y=bx+CTz+BxTx+u, where z, x, and u are independent, C remains unchanged compared with (3), Eu=0 , E‖u‖2<∞ , and bx=b+BTΣδΣx−1μ−ΣϵδΣx−1μ, BxT=BTΣξΣx−1+ΣϵδΣx−1.

Assume additionally the following condition:

Either Σe or Σ11−Σ12Σ22−1Σ12T is positive definite.

 Then the error term u in (6) has a positive definite covariance matrix, Σu .

(a) Introduce the jointly Gaussian vectors x(1)=ξϵ,x(2)=x. We have μ(1):=Ex(1)=μ0,μ(2):=Ex(2)=μ;Covx(1)=Σ11,Covx(2)=Σ22, which is positive definite by assumption (ii), Ex(1)x(2)T=Σ12, where the matrices Σ11 , Σ12 , Σ22 are given in (5). Now, according to Theorem 2.5.1 [1] the conditional distribution of x(1) given x(2) is x(1)x(2)∼Nμ1|2,V1|2,μ1|2=μ1|2(x(2))=μ(1)+Σ12Σ22−1x(2)−μ(2)=ΣδΣx−1μ+ΣξΣx−1xΣϵδΣx−1(x−μ),V1|2=Σ11−Σ12Σ22−1Σ12T.

Hence (ξT,ϵT)T−μ1|2(x)=:(γ1T,γ2T)T is uncorrelated with x and has the Gaussian distribution N0,V1|2 . Therefore, ξ=ΣδΣx−1μ+ΣξΣx−1x+γ1, ϵ=ΣϵδΣx−1x−μ+γ2. Substitute (10) and (11) into (3) and obtain the desired relations (6)–(8) with u=e+BTγ1+γ2. Here (z,e,x) and a couple (γ1,γ2) are independent, hence (z,x,u) are independent as well. This implies the statement (a).

(b) We have Cov(u)=Σe+CovBTγ1+γ2=:Σu. If Σe>0 0$]]> then Σu≥Σe>0 0$]]>, thus, Σu>0 0$]]>; and if V1|2>0 0$]]> then Σu≥CovBTγ1+γ2>0 0$]]>, thus, Σu>0 0$]]>. This accomplishes the proof of Lemma 1.  □

As a particular case take a model with a univariate response and univariate regressor ξ.

Consider the model (3), (4) with  d=m=1 . Assume conditions (i), (iii), and (iv). Suppose also that Σz>0,Σϵ>0,Σδ>0,Corr(ϵ,δ)<1. 0,\hspace{2em}{\Sigma _{\epsilon }}>0,\hspace{2em}{\Sigma _{\delta }}>0,\hspace{2em}\left|\operatorname{Corr}(\epsilon ,\delta )\right|<1.\]]]>

Then expressions (6)–(8) hold true, where the error term u has a positive variance σu2=Σu .

First suppose that Σξ>0 0$]]>. According to Lemma 1, it is enough to check that V1|2 given in (9) is positive definite.

A direct computation shows that V1|2=1σx2σξ2σδ2−σξ2σϵδ−σξ2σϵδσϵ2σx2−σϵδ2=:Vσx2. Here in the scalar case we write σξ2=Σξ , σδ2=Σδ , σϵδ=Σϵδ , etc. The matrix V is positive definite, because σξ2σδ2>0 0$]]> and detV=σδ2σx2σϵ2σδ2−σϵδ2>0 0\]]]> due to condition (13).

Now, suppose that Σξ=0 . Then ξ=μ almost surely. With some computations, it can be shown that u=e+ϵ−σϵδσδ−2δ almost surely, whence σu2=σe2+σϵ2−σϵδ2σδ−2>0 0$]]>. Lemma 2 is proved.  □

 Individual prediction

Now, consider independent copies of the multivariate model (3), (4): yi,zi,ξi,ei,ϵi,xi,δi,i=1,…,n. Based on the observations yi,zi,xi,i=1,…,n, and for given z0 , x0 , we want to estimate the individual predictor yˆ0 presented in (1) and the mean predictor ηˆ0 presented in (2).

Assume conditions (i) to (iv) and suppose that all model parameters are unknown. Lemma 1 implies the expansion (6) with  Eu=0 . All the underlying random vectors have finite 2nd moments, hence yˆ0:=bx+CTz0+BxTx0 is the best mean squared error predictor of  y0 . Since it is unfeasible, we have to estimate the coefficients bx , C and Bx using the sample (14). The OLS estimator bˆx,Cˆ,Bˆx minimizes the penalty function Q(b,C,B):= ∑i=1nyi−b−CTzi−BTxi2,b∈R,C∈Rq×d,B∈Rm×d.

Let bar denote the average over i=1,…,n , e.g., y¯=1n ∑i=1nyi, and Suv denote the sample covariance matrix of u and v variables, e.g., Sxy=1n ∑i=1nxi−x¯yi−y¯T,Sxx=1n ∑i=1nxi−x¯xi−x¯T, etc. The OLS estimator can be computed from the relations [11] y¯=bˆx+CˆTz¯+BˆxTx¯, CˆBˆx=Srr+Sry,r:=zx. Hereafter A+ is the pseudo-inverse of a square matrix A; see the properties of A+ in [8]. The corresponding OLS predictor is y˜0:=bˆx+CˆTz0+BˆxTx0.

Assume conditions (i) to (iv). Then y˜0 presented in (19) is a strongly consistent estimator of the best predictor yˆ0 , i.e. y˜0→yˆ0a.s. asn tends to infinity. Moreover, ∀τ>0,P(y˜0−yˆ0‖>τz0,x0→0a.s. asn→∞. 0,\hspace{1em}\operatorname{\mathbf{P}}(\left.\left\| {\tilde{y}_{0}}-{\hat{y}_{0}}\| >\tau \right|{z_{0}},{x_{0}}\right)\to 0\hspace{1em}\textit{a.s. as}\hspace{2.5pt}n\to \infty .\]]]>

By Strong Law of Large Numbers we have a.s. as  n→∞ : Srr→block‐diagΣz,Σx>0,Sry→Covz,yCovx,y=Σz·CΣx·Bx,CˆBˆx→Σz−1Σz·CΣx−1Σx·Bx=CBx. 0,\\ {} \displaystyle {S_{ry}}\to \left(\substack{\operatorname{\mathbf{Cov}}\left(z,y\right)\\ {} \operatorname{\mathbf{Cov}}\left(x,y\right)}\right)=\left(\substack{{\Sigma _{z}}\cdot C\\ {} {\Sigma _{x}}\cdot {B_{x}}}\right),\\ {} \displaystyle \left(\substack{\hat{C}\\ {} {\hat{B}_{x}}}\right)\to \left(\substack{{\Sigma _{z}^{-1}}{\Sigma _{z}}\cdot C\\ {} {\Sigma _{x}^{-1}}{\Sigma _{x}}\cdot {B_{x}}}\right)=\left(\substack{C\\ {} {B_{x}}}\right).\end{array}\]]]> This convergence, relation (17) and the a.s. convergence of the sample means imply that bˆx→bxa.s. Now, both statements of Theorem 1 follow from (19) and (15).  □

It is interesting to construct an asymptotic confidence region for the response y0 based on the OLS predictor. Assume (i) to (iv). It holds Covy0−yˆ0z0,x0=Cov(u0)=Σu, see (12). Introduce the estimator Σˆu=1n ∑i=1nyi−bˆx−CˆTzi−BˆxTxiyi−bˆx−CˆTzi−BˆxTxiT.

Suppose that conditions (i) to (iv) hold. Fix the confidence probability 1−α .

Assume additionally (v) and define Eα=h∈Rd:Σˆu+1/2h−y˜02≤dα. Then lim infn→∞Py0∈Eαz0,x0≥1−α.

Let the model (3)–(4) be purely normal, i.e. z is normally distributed and e=0 . Assume additionally that the matrix (9) is nonsingular. Define Dα=h∈Rd:Σˆu+1/2h−y˜02≤χdα2, where χdα2 is an upper α-quantile of  χd2 distribution, i.e. Pχd2>χdα2=α {\chi _{d\alpha }^{2}}\right)=\alpha $]]>. Then limn→∞Py0∈Dαz0,x0=1−α.

If bx , C, and Bx were known, then we could approximate Σu as follows: 1n ∑i=1nuiuiT→Σua.s. asn→∞,ui:=yi−bx−CTzi−BxTxi. Since uiuiT is a quadratic function of the coefficients bx , C, Bx , and the OLS estimators of those coefficients are strongly consistent, the convergence (25) remains valid if we replace all ui with the residuals uˆi:=yi−bˆx−CTzi−BˆxTxi. Hence Σˆu→Σua.s. asn→∞.

(a) Under (v), Σu is nonsingular by Lemma 1(b). It holds PΣu−1/2y0−yˆ02>dαz0,x0≤αEΣu−1/2·u2d=α. \frac{d}{\alpha }\right|{z_{0}},{x_{0}}\right)\le \alpha \frac{\operatorname{\mathbf{E}}{\left\| {\Sigma _{u}^{-1\hspace{-0.1667em}/2}}\cdot u\right\| ^{2}}}{d}=\alpha .\]]]> Since the relations (20) and (26) hold true, the relations (22), (21) follow.

(b) Again, in this purely normal model the matrix Σu is nonsingular; conditional on  z0 and x0 , the difference y0−yˆ0=u0 has the normal distribution N0,Σu . Then PΣu−1/2y0−yˆ02>χdα2z0,x0=α. {\chi _{d\alpha }^{2}}\right|{z_{0}},{x_{0}}\right)=\alpha .\]]]> Since the relations (26) and (20) hold true, the relations (24), (23) follow.  □

For the univariate model with d=m=1 , assume the conditions of Lemma 2. Then relations (22), (21) hold true. If additionally z=0 and e=0 then relations (24) and (23) are valid.

 Mean prediction

Still consider the model (3), (4) under conditions (i) to (iv). We want to estimate the mean predictor ηˆ0 presented in (2). We have ηˆ0=yˆ0−Ee0z0,x0−Eϵ0z0,x0,Ee0z0,x0=Ee0=0, and by (11), Eϵ0z0,x0=Eϵ0x0=ΣϵδΣx−1x0−μ. Thus, ηˆ0=yˆ0−ΣϵδΣx−1x0−μ.

Based on observations (14), strongly consistent and unbiased estimators of μ and Σx are as follows: μˆ=x¯=1n ∑i=1nxi, Σˆx=1n−1 ∑i=1nxi−x¯xi−x¯T.

Assume conditions (i) to (iv) and suppose that Σϵδ is the only model parameter which is known. Consider the estimators (19), (28), and (29). Then η˜0:=y˜0−ΣϵδΣˆx−1x0−μˆ is a strongly consistent estimator of the mean predictor (2), and moreover ∀τ>0,Pη˜0−ηˆ0>τz0,x0→0a.s. asn→∞. 0,\hspace{1em}\operatorname{\mathbf{P}}\left(\left.\left\| {\tilde{\eta }_{0}}-{\hat{\eta }_{0}}\right\| >\tau \right|{z_{0}},{x_{0}}\right)\to 0\hspace{1em}\textit{a.s. as}\hspace{2.5pt}n\to \infty .\]]]>

The statement follows from relation (27), Theorem 2, and the strong consistency of the estimators  μˆ and Σxˆ .  □

Notice that more model parameters should be known in order to construct a confidence region for  η0 around η˜0 .

 Prediction in a polynomial EIV model Model and main assumptions

For a fixed and known k≥2 , consider a polynomial EIV model (structural case): y=cTz+β0+βTξ,ξ2,…,ξkT+e+ϵ, x=ξ+δ. Here the random variable (r.v.) y is the response variable; the random vector z is the observable covariate distributed in  Rq , r.v. ξ is the unobservable covariate; x is the surrogate data observed instead of ξ; e is the random error in the equation, ϵ and δ are the measurement errors in the response and in the latent covariate; c∈Rq , β0∈R and β=β1,…,βkT∈Rk contain unknown regression parameters; ϵ and δ can be correlated.

Such models are studied, e.g., in [3, 6] and applied, for instance, in econometrics. Let us introduce the model assumptions.

The random variables ξ, e and random vectors z, ϵ,δT are independent, with finite 2nd moments; the random variables ϵ and δ can be correlated.

The covariance matrix Σz:=Cov(z) is nonsingular, and σx2:=Var(x)>0 0$]]>.

The errors e, ϵ and δ have zero mean.

The errors ϵ, δ and ξ are jointly Gaussian.

We see that assumptions (a) to (d) are similar to conditions (i) to (iv) imposed on the multivariate linear model, but now the response and latent covariate are real valued.

 Regression <italic>y</italic> on <italic>z</italic> and <italic>x</italic>

Let us denote σϵδ=Eϵδ,μ=Ex,σξ2=Var(ξ),σe2=Var(e),σδ2=Var(δ).

Assume conditions (a) to (d). Then the response variable (30) admits the representation y=cTz+β0x+βxTx,x2,…,xkT+u, where z and (x,u)T are independent, the vector c remains unchanged compared with (30), Eux=0 , Eu2x<∞ , and β0x∈R , βx∈Rk are transformed (nonrandom) parameters of the polynomial regression.

In the new notation, we have from (10) and (11): ξ=σδ2σx−2μ+σξ2σx−2x+γ1=:a+Kx+γ1, ϵ=σϵδσx−2(x−μ)+γ2=:b+fx+γ2, where z, x and (γ1,γ2)T are independent, and (γ1,γ2)T has the Gaussian distribution N0,V1|2 .

Now, substitute (34) and (35) into (30) and get y=cTz+β0+ ∑j=1kβj(a+Kx+γ1)j+b+fx+e+γ2,y=cTz+β0+ ∑j=1kβj ∑p=0jjp(a+Kx)j−pEγ1px+b+fx+u, u=e+γ2+ ∑j=1kβj ∑p=1jjp(a+Kx)j−pγ1p−Eγ1px. It holds Eux=0 , Eu2x<∞ , and relations (36)–(37) imply the statement.  □

 Individual and mean prediction

We consider independent copies of the polynomial model (30)–(31): yi,zi,ξi,ei,ϵi,xi,δi,i=1,…,n. Based on observations (14) and for given z0 , x0 , we want to estimate the individual predictor yˆ0 and the mean predictor ηˆ0 for the polynomial model.

Assume conditions (a) to (d) and suppose that all model parameters are unknown. Lemma 3 implies the expansion (33) with  Eu|x,z=0 . All the underlying r.v.’s and the random vector z have finite 2nd moments, hence yˆ0:=cTz0+β0x+βxTx0,x02,…,x0kT is the best mean squared error predictor of  y0 . We estimate the coefficients c, β0x and βx using the sample (14) from the polynomial model. The OLS estimator minimizes the penalty function Qc,β0,β:= ∑i=1nyi−cTzi−β0−βTxi,xi2,…,xikT2, c∈Rq , β0∈R , β∈Rk . The OLS estimator can be computed by relations similar to (17)–(18): y¯=cˆTz¯+βˆ0x+βˆxTx‾,x2‾,…,xk‾T,cˆβˆx=Srr+Sry,r:=(zT,x,…,xk)T; the sample covariance matrices Srr and Sry are defined in (16). The corresponding OLS predictor is y˜0:=cˆTz0+βˆ0x+βˆxTx0,x02,…,x0kT.

Assume conditions (a) to (d). Then y˜0 presented in (39) is a strongly consistent estimator of the individual predictor yˆ0 . Moreover, ∀τ>0,Py˜0−yˆ0>τz0,x0→0a.s. asn→∞. 0,\hspace{0.2778em}\operatorname{\mathbf{P}}\left(\left.\left\| {\tilde{y}_{0}}-{\hat{y}_{0}}\right\| >\tau \right|{z_{0}},{x_{0}}\right)\to 0\hspace{1em}\textit{a.s. as}\hspace{2.5pt}n\to \infty .\]]]>

Following the lines of the proof of Theorem 2, it is enough to check the strong consistency of the estimators cˆ and βˆx . We have a.s. as n→∞ : Srr→Cov(r)=block‐diagΣz,D,D:=Cov(x,x2,…xk), Sry→Cov(r,y)=Σz·cDβx. By conditions (b) and (d), x is a nondegenerate Gaussian r.v., therefore, r.v.’s 1,x,…,xk are linearly independent in the Hilbert space L2Ω,P of square integrable r.v.’s, and the covariance matrix D is nonsingular. Relations (38), (40), and (41) imply that a.s. as n→∞ cˆβˆx→Σz−1ΣzcD−1Dβx=cβx. And the statements of Theorem 4 follow.  □

Similarly to Theorem 3 one can construct a consistent estimator of the mean predictor (2) in the polynomial EIV model. The strongly consistent estimator of μ is given in (28) and the one of  σx2 is constructed similarly to (29): σˆx2=1n−1 ∑i=1nxi−x¯2.

Assume conditions (a) to (d) and suppose that σϵδ defined in (32) is the only parameter which is known in the model (30), (31). Consider the estimators (39), (28), and (42). Then η˜0:=y˜0−σϵδσˆx−2(x0−μˆ) is a strongly consistent estimator of the mean predictor (2), and moreover, ∀τ>0,Pη˜0−ηˆ0>τz0,x0→0a.s. asn→∞. 0,\hspace{1em}\operatorname{\mathbf{P}}\left(\left.\left\| {\tilde{\eta }_{0}}-{\hat{\eta }_{0}}\right\| >\tau \right|{z_{0}},{x_{0}}\right)\to 0\hspace{1em}\textit{a.s. as}\hspace{2.5pt}n\to \infty .\]]]>

Confidence interval for response in quadratic model

Consider a quadratic EIV model y=β0+β1ξ+β2ξ2+e, x=ξ+δ. It is a particular case of the model (30), (31) with  k=2 , z=0 and ϵ=0 .

We use notations (32). Our conditions are similar to (a)–(d), but we assume additionally that the reliability ratio K:=σξ2σx2 is separated away from zero. Thus, assume the following conditions.

The random variables ξ, e and δ are independent; ξ and δ are Gaussian; e and δ have zero mean and σe2<∞ ; σx2>0 0$]]>.

Model parameters are unknown, but a lower bound K0 for the reliability ratio (45) is given, with  0<K0≤1/2 .

Consider indepedent copies of the quadratic model yi,ξi,ei,xi,δi,i=0,1,…,n. Based on observations (yi,xi),i=1,…,n , and for a given x0 , we can construct the OLS predictor y˜0 , see (39), for  y0 with  k=2 , z0=0 . Now, we show the way how to construct an asymptotic confidence interval for  y0 . (In a similar way this can be done for a polynomial EIV model of higher order.)

First we write down the representation (36), (37). Denote mx=Eξx=Kx+1−Kμ. We have with independent mx and γ: ξ=mx+γ,γ∼N0,Kσδ2. Then y=β0+β1(mx+γ)+β2(mx+γ)2+e==β0+β1mx+β2mx2+Kσδ2+u=:yˆ+u, u=e+(β1+2mxβ2)γ+β2γ2−Kσδ2. Here Eux=0 . From (3.4) we get that the best prediction is yˆ=β0x+β1x·x+β2x·x2,β1x=β1K+2β2K(1−K)μ,β2x=β2·K2. Those coefficients can be estimated using the strongly consistent OLS estimator, cf. (38), βˆ1xβˆ2x=Srr+Sry,r:=(x,x2)T. The OLS estimator βˆ0x satisfies y¯=βˆ0x+βˆ1xx¯+βˆ2xx2‾, and the OLS predictor of  y0 is equal to y˜0=βˆ0x+βˆ1xx0+βˆ2xx02.

To construct a confidence interval for  y0 , we have to bound the conditional variance of u given x0 . From (49) we have Varux=σϵ2+(β1+2mxβ2)2Kσδ2+β22·2Kσδ22, where 2Kσδ22=Var(γ2) . Denote mu2=EVarux. It holds a.s. asn→∞ : 1n ∑i=1nyi−β0x−β1xxi−β2xxi22→mu2. Therefore, we have a.s. asn→∞ : mˆu2:=1n ∑i=1nyi−βˆ0x−βˆ1xxi−βˆ2xxi22→mu2. We have to bound the difference Varux−mu2=4Kσδ2β22K2·F(K,x,μ)+β1β2K(x−μ),F(k,x,μ):=x2−μ2−σx2+2K(1−K)μ(x−μ). Here we used the relations mx−Emx=K(x−μ),mx2−Emx2=K2−μ2−σx2+2K(1−K)μ(x−μ). Next, we express (52) through βix rather than βi . Using (50) we get: σδ2=σx2(1−K),Varux−mu2=4(1−K)σx2·β2x2KF(K,x,μ)−2(1−K)Kμ(x−μ)++4(1−K)σx2β1xβ2x·x−μK≤41K0−1σx2·G(x,μ,σx2,β1x,β2x),G(x,μ,σx2,β1x,β2x)=β2x2[x2−μ2−σx2++2μ(x−μ)−(1−K0)21+1K0]+β1xβ2x(x−μ)+. Here A+:=max(A,0) , A−:=−min(A,0) , A∈R . Finally, Varux≤mu2+41K0−1σx2G(x,μ,σx2,β1x,β2x).

We are ready to construct a confidence interval for  y0 .

For the model (43)–(44), assume conditions (e) and (f). Fix the confidence probability 1−α . Define Iα={h∈R:h−y˜0≤α−1/2××mˆu2+41K0−1σˆx2G(x0,μˆ,σˆx2,βˆ1x,βˆ2x)+1/2}, where mˆu2 , σˆx2 , μˆ , βˆ1x and βˆ2x are strongly consistent estimators of the corresponding parameters; the estimators were presented above. Then lim infn→∞Py0∈Iαx0≥1−α.

It holds for  t>0 0$]]>: Py0−yˆ0>tx0≤Varux0t2≤α t\right|{x_{0}}\right)\le \frac{\operatorname{\mathbf{Var}}\left(\left.u\right|{x_{0}}\right)}{{t^{2}}}\le \alpha \]]]> if t is selected such that t≥α−1/2Varux01/2 . Now, the statement follows from the inequality (53) and the consistency of  y˜0 , mˆu2 , σˆx2 , μˆ , βˆ1x and βˆ2x .  □

 Prediction in other EIV models

The OLS predictor y˜ approximates the best mean squared error predictor yˆ presented in (1) not only in the plynomial EIV model. Let us consider the model with exponential regression function y=βeλξ+e,x=ξ+δ, where the real numbers β and λ are unknown regression parameters, and assume condition (e) from Section 3.4. Using expansion (47)–(46), we get y=βxexpλx·x+u=:yˆ+u, βx=βeλ(1−K)μ·Eeλγ,λx=Kλ,Eeλγ=expλ2Kσδ22,u=βxeλx·x(eλγ−Eeλγ).

Under mild conditions, the OLS predictor y˜0:=βˆxexpλˆx·x0 is a strongly consistent estimator of  yˆ0 , where βˆx and λˆx are the OLS estimators of the regression parameters in the model (54).

Similar conclusion can be made for the trigonometric model y=a0+ ∑k=1makcoskωξ+bksinkωξ+e,x=ξ+δ, where ak,0≤k≤m , bk,1≤k≤m , and ω>0 0$]]> are unknown regression parameters.

Finally, we give an example of the model, where the OLS predictor does not approximate the best mean squared error predictor. Let y=β|ξ+a|+e,x=ξ+δ, where the real numbers β and a are unknown regression parameters, and assume condition (e) from Section 3.4; suppose also that σξ2 and σδ2 are positive.

For  γ0∼N(0,1) , evaluate F(a):=E|γ0+a|=2ϕ(a)+a(2Φ(a)−1),a∈R, where ϕ and Φ are the pdf and cdf of  γ0 . Then the best mean squared error predictor is as follows: yˆ=Eyx=βE[|a+Kx+(1−K)μ+σδKγ0||x]==βxFkx·x+bx,kx>0,βx∈R,bx∈R,βx=βσδK,kx=Kσδ,bx=a+(1−K)μσδK. 0,\hspace{2.5pt}{\beta _{x}}\in \mathbb{R},\hspace{2.5pt}{b_{x}}\in \mathbb{R},\end{aligned}\\ {} \displaystyle {\beta _{x}}=\beta {\sigma _{\delta }}\sqrt{K},\hspace{2.5pt}{k_{x}}=\frac{\sqrt{K}}{{\sigma _{\delta }}},\hspace{2.5pt}{b_{x}}=\frac{a+(1-K)\mu }{{\sigma _{\delta }}\sqrt{K}}.\end{array}\]]]> The LS estimators kˆx , βˆx and bˆx of  kx , βx and bx minimize the penalty function QLS(k,β,b):= ∑i=1nyi−βF(kxi+b)2. Under mild additional conditions, the LS estimators are strongly consistent, and the LS predictor y˜0:=βxFbˆx·x0+bˆx converges a.s. to  yˆ0=Ey0x0=βxFbx·x0+bx as the sample size grows. Notice that for this model (57), the OLS predictor βˆx0+aˆ needs not to converge in probability to  yˆ0 , where the OLS estimators βˆ and aˆ minimize the penalty function QOLS(β,a):= ∑i=1nyi−β|xi+a|2.

Conclusion

We considered structural EIV models with the classical measurement error. We gave a list of models where the OLS predictor of response  y0 converges with probability one to the best mean squared error predictor yˆ0=Ey0z0,x0 . In such models, a functional dependence yˆ0=yˆ0(z0,x0) belongs to the same parametric family as the initial regression function η0(z0,ξ0)=Ey0z0,ξ0 . Such a situation looks exceptional for nonlinear models, and we gave an example of model (57), where the OLS predictor does not perform well.

We dealt with both the mean and individual prediction. They coincide in the case of nondifferential errors, where it is known that the errors in response and in covariates are uncorrelated. Otherwise, to construct the mean prediction, one has to know the covariance of the errors.

In linear models, we managed to construct an asymptotic confidence region for response around the OLS prediction, under totally unknown model parameters. In the quadratic model, we did it under the known lower bound of the reliability ratio. The procedure can be expanded to polynomial models of higher order.

Notice that in linear models without intercept and in incomplete polynomial models (like, e.g. y=β0+β2ξ2+e , x=ξ+δ ), a prediction with (z,x) naively substituted for  (z,ξ) in the regression of y on  (z,x) can have huge prediction errors. As stated in [2, Section 2.6], predicting y from  (z,x) is merely a matter of substituting known values of x and z into the regression model for y on  (z,x) . We can add that, in nonlinear EIV models, the corresponding error v=y−Eyz,x has the variance depending on x, i.e., the regression of y on  (z,x) is heteroskedastic; this should be taken into account in order to construct a confidence region for y in a proper way.

Finally, we make a caveat for practitioners. Consistent EIV regression parameter estimators are useful especially for prediction if the observation errors for the predicted subject differ from those in the data used for the model fitting. This is usually the case when the model is fitted by some experimental data while the prediction is made for a real world subject. The idea to use inconsistent OLS estimators for prediction in this case is not good.

 Acknowledgments

We are grateful to Dr. S. Shklyar (Kyiv) for fruitful discussions.

 References Anderson, T.W.: An Introduction to Multivariate Statistical Analysis. Wiley Publications in Statistics. John Wiley & Sons, Inc. and Chapman & Hall, Ltd. (1958). MR0091588 Carroll, R.S., Ruppert, D., Stefanski, L.A., Crainiceanes, C.M.: Measurement Error in Nonlinear Models: A Modern Perspective, Second Edition. Monogr. Stat. Appl. Probab., vol. 105. Chapman and Hall/CRC (2006). MR2243417. https://doi.org/10.1201/9781420010138 Cheng, C.-L., Schneeweiss, H.: Polynomial regression with errors in the variables. J. R. Stat. Soc. 60(1), 189–199 (1998). MR1625632. https://doi.org/10.1111/1467-9868.00118 Cheng, C.-L., Van Ness, J.W.: Statistical Regression with Measurement Error. Kendall’s Library of Statistics 6. Arnold (1999). MR1719513. https://doi.org/10.1002/1097-0258(2000815)19:15 Gujarati, D.N., Porter, D.C., Gunaseker, S.: Basic Econometrics, 5th Edn. MCGraw-Hill (2017) Kukush, O.G., Tsaregorodtsev, Y.V.: Convergence of estimators in a polynomial functional model with measurement errors. Theory Probab. Math. Stat. 92, 79–88 (2016). MR3553428. https://doi.org/10.1090/tpms/984 Mynbaev, K., Martins-Filho, C.: Consistency and asymptotic normality for a nonparametric prediction under measurement errors. J. Multivar. Anal. 139, 166–188 (2015). MR3349485. https://doi.org/10.1016/j.jmva.2015.03.003 Seber, G.A.F., Lee, A.S.: Linear Regression Analysis, 2nd Edn. Wiley Series in Probability and Statistics. John Wiley & Sons (2003). MR1958247. https://doi.org/10.1002/9780471722199 Shklyar, S.: Consistency of the total least squares estimator in the linear errors-in-variables regression. Mod. Stoch. Theory Appl. 5(3), 247–295 (2018). MR3868542. https://doi.org/10.15559/18-vmsta104 Söderström, T.: Errors-in-Variables Methods in System Identification. Communications and Control Engineering Series. Springer (2018). MR3791479. https://doi.org/10.1007/978-3-319-75001-9 Van Huffel, S., Vandewalle, J.: The Total Least Squares Problem. Frontiers Appl. Math., vol. 9. MR1118607. https://doi.org/10.1137/1.9781611971002 Wansbeek, T., Meijer, E.: Measurement Error and Latent Variables in Econometrics. Advanced Textbooks in Economics. North Holland Publishing Co. (2000). MR1804397. https://doi.org/10.1002/9780470996249