In this paper, we discuss a structural linear regression technique in the context of model of mixture with varying concentrations (MVC). MVC means that the observed subjects belong to M different subpopulations (mixture components). The true numbers of components to which the subjects Oj , j=1,…,N , belong, say, κj=κ(Oj) , are unknown, but we know the probabilities pj;Nk=defP{κj=k},k=1,…,M (mixing probabilities or concentrations of the mixture components). MVC models arise naturally in the description of medical, biologic, and sociologic data [1, 8, 9, 12]. They can be considered as a generalization of finite mixture models (FMM). Classical theory of FMMs can be found in monographs [10, 13].

Let ξ(O)=(Y(O),X1(O),…,Xd(O))T be a vector of observed features (random variables) of a subject O. We consider the following linear regression model for these variables: (1) Y(O)= ∑i=1dbi(κ(O))Xi(O)+ε(O), where b(m)=(b1(m),…,bd(m))T are nonrandom regression coefficients for the mth component, ε(O) is an error term, which is assumed to be zero mean and conditionally independent of the regressors vector X(O)=(X1(O),…,Xd(O))T given κ(O) .

Note. We consider a subject O as taken at random from an infinite population, so it is random in this sense. The vector of observed variables ξ(O) can be considered as a random vector even for a fixed O.

Our aim is to estimate the vectors of regression coefficients b(k) , k=1,…,M , by the observations ΞN=(ξ1,…,ξN) , where ξj=ξ(Oj) . We assume that (κj,ξj) are independent for different j.

A statistical model similar to MVC with (1) is considered in [5], where a parametric model for the conditional distributions of ε(O) given κ(O) is assumed. For this case, maximum likelihood estimation is proposed in [5], and a version of EM-algorithm is developed for numerical computation of the estimates.

In this paper, we adopt a nonparametric approach assuming no parametric models for ε(O) and X(O) distributions. Nonparametric and semiparametric technique for MVC was developed in [6, 7, 4]. We use the weighted empirical moment technique to derive estimates for the regression coefficients and then obtain conditions of consistency and asymptotic normality of the estimates. These results are based on general ideas of least squares [11] and moment estimates [3].

The rest of the paper is organized as follows. In Section 2, we recall some results on nonparametric estimation of functional moments in general MVC. The estimates are introduced, and conditions of their consistency and asymptotic normality are presented in Section 3. Section 4 contains proofs of the statements of Section 3. Results of computer simulations are presented in Section 5.

Let us start with some notation and definitions. We denote by Fm the distribution of ξ(O) for O belonging to the mth component of the mixture, that is, Fm(A)=defP{ξ(O)∈A|κ(O)=m} for all measurable sets A. Then by the definition of MVC (2) P{ξj∈A}= ∑k=1Mpj;NkFk(A). In the asymptotic statements, we will consider the data Ξn=(ξ1,…,ξN) as an element of (imaginary) series of data Ξ1,Ξ2,…,ΞN,… in which no link between observations for different N is assumed. So, in formal notation, it should be more correct to write ξj;N instead of ξj , but we will drop the subscript N when it is insignificant.

We consider an array of all concentrations for all data sizes p=def(pj;Nk,k=1,…,M;j=1,…,N;N=1,2,…). Its subarrays pNk=def(p1;Nk,…,pN;Nk)Tandpj;N=(pj;N1,…,pj;NM)T are considered as vector columns, and pN=(pj;Nk,j=1,…,N;k=1,…,M) as an N×M -matrix. We will also consider a weight array a of the same structure as p with similar notation for its subarrays.

By the angle brackets with subscript n we denote the averaging by j=1,…,n : ⟨ak⟩N=def1N ∑j=1Naj;Nk. Multiplication, summation, and other operations in the angle brackets are made elementwise: ⟨akpm⟩N=1N ∑j=1Naj;Nkpj;Nm;⟨(ak)2⟩N=1N ∑j=1N(aj;Nk)2. We define ⟨pm⟩=deflimN→∞⟨pm⟩N if this limit exists. Let ΓN=(⟨plpm⟩N)l,m=1M=1NpNTpN be an M×M matrix, and γlm;N be its (l,m) th minor. The matrix ΓN can be considered as the Gramian matrix of vectors (p1,…,pM) in the inner product ⟨pipk⟩N , so that it is nonsingular if these vectors are linearly independent.

In what follows, ⟶P means convergence in probability, and ⟶W means weak convergence.

Assume now that model (2) holds for the data ΞN . Then the distribution Fm of the mth component can be estimated by the weighted empirical measure (3) Fˆm;N(A)=def1N ∑j=1Naj;Nm1{ξj∈A}, where (4) aj;Nm=1detΓN ∑k=1M(−1)k+mγmk;Npj;Nk. It is shown in [8] that if ΓN is nonsingular, then Fˆm;N is the minimax unbiased estimate for Fm . The consistency of Fˆm;N is demonstrated in [6] (see also [8]).

Consider now functional moment estimation based on weighted empirical moments. Let g:Rd+1→Rk be a measurable function. Then to estimate g¯(m)=E[g(ξ(O))|κ(O)=m]=∫g(x)Fm(dx), we can use gˆ;N(m)=∫g(x)Fˆm;N(dx)=1N ∑j=1Naj;Nmg(ξj).

Then gˆ;N(m)⟶Pg¯(m) as N→∞ .

This lemma is a simple corollary of Theorem 4.2 in [8]. (See also Theorem 3.1.1 in [7]).

Then N(gˆ;N(m)−g¯(m))⟶WN(0,Σ). For univariate gˆ;N(m) , the statement of the lemma is contained in Theorem 4.2 from [8] (or Theorem 3.1.2 in [7]). The multivariate case can be obtained from the univariate one applying the Cramér–Wold device (see [2], p. 382).

In view of Lemma 1, we expect that, under suitable assumptions, Jm;N(b)=def1N ∑j=1Naj;Nm(Yj;N− ∑i=1dbiXj:Ni)2=∫(y− ∑i=1dbixi)2Fˆm;N(dy,dx1,…,dxd) converges to ∫(y− ∑i=1dbixi)2Fm(dy,dx1,…,dxd)=E[(Y(O)− ∑i=1dbiXi(O))2|κ(O)=m]=defJm;∞(b) as N→∞ .

Since Jm;∞(b) attains its minimal value at b(m) , it is natural to suggest the argmin of Jm;N(b) as an estimate for b(m) . If the weights am were positive, then this argmin would be (5) bˆ;N(m)=def(XTAX)−1XTAY, where X=def(Xji)j=1,…,N;i=1,…,d is the N×d matrix of observed regressors, Y=def(Y1,…,YN)T is the vector of observed responses, and A=defdiag(a1;Nm,…,aN;Nm) is the diagonal weight matrix for estimation of mth component. (Obviously, A depends on m, but we do not show it explicitly by a subscript since the number m of the component for which bm is estimated will be further fixed.)

Generally speaking, by (4) aj;Nm must be negative for some j, so bˆ;N(m) is not necessarily an argmin of Jˆm;N(b) . But we will take bˆ;N(m) as an estimate for bm and call it a modified least-squares estimate for bm in MVC model (MVC-LS estimate).

Let D(k)=defE[X(O)XT(O)|κ(O)=k] be the matrix of second moments of the regressors for subjects belonging to the kth component. Denote the variance of the kth component’s error term by (σ(k))2=E[(ε(O))2|κ(O)=k]. (Recall that E[(ε(O))|κ(O)=k]=0 ). In what follows, we assume that these moments and variances exist for all components.

Then bˆ;N(m)⟶Pb(m) as N→∞ .

Note. Assumption 3 can be weakened. Applying Theorem 4.2 from [8], we can show that bˆ;N(m) is consistent if the vector pNm is asymptotically linearly independent from the vectors pNi , i≠m , as N→∞ . To avoid complexities in this presentation, we do not formulate the strict meaning of this statement.

Denote Dik(s)=defE[Xi(O)Xk(O)|κ(O)=s] , Lik(s)=def(E[Xi(O)Xk(O)Xq(O)Xl(O)|κ(O)=s])l,q=1d,Mik(s,p)=def(Dil(s)Dkq(p))l,q=1d.

Then N(bˆ;N(m)−b(m))⟶WN(0,V) , where (6) V=defD−1ΣD−1 with (7) Σ=(Σik)ik=1d,Σik= ∑s=1M⟨(am)2ps⟩(Dik(s)(σ(s))2+(b(s)−b(m))TLik(s)(b(s)−b(m)))− ∑s=1M ∑p=1M⟨(am)2pspp⟩(b(s)−b(m))TMik(s,p)(b(p)−b(m)).

To assess the accuracy of the asymptotic results from Section 3, we performed a small simulation study. We considered a two-component mixture ( M=2 ) with mixing probabilities pj;N1=j/N and pj;N2=1−pj;N1 . For each subject, there were two observed variables X and Y, which were simulated based on the simple linear regression model Yj=b0κj+b1κjXj(κj)+εj(κj), where κi is the number of component the jth observation belongs to, Xj(1) was simulated as N(1,1) , Xj(2) as N(2,2.25) , and εj(k) were zero-mean Gaussians with standard deviations 0.01 for the first component and 0.05 for the second one. The values of the regression coefficients were b01=3 , b11=0.5 , b02=−2 , b12=1 .

The means and covariances of the estimates were calculated over 2000 replications.

The results of simulation are presented in Table 1. The true values of parameters and asymptotic covariances are placed in the last rows of the tables.

The presented data show good concordance with the asymptotic theory for n>1000 1000$]]>.

We considered a modification of least-squares estimators for linear regression coefficients in the case where observations are obtained from a mixture with varying concentrations. Conditions of consistency and asymptotic normality of the estimators were derived, and dispersion matrices were evaluated. The results of simulations confirm good concordance of estimators covariances with the asymptotic formulas for sample sizes larger then 1000 observations.

In real-life data analysis, concentrations (mixing probabilities) are usually not known exactly but estimated. So, to apply the proposed technique, we also need to analyze sensitivity of the estimates to perturbations of the concentrations model. (We are thankful to the unknown referee for this observation). It is worth noting that performance of these estimates will be poor if the true concentrations of the components are nearly linearly dependent ( detΓN≈0 ). We also expect stability of the estimates w.r.t. concentration perturbations if detΓN is bounded away from zero. More deep analysis of sensitivity will be a part of our further work.