Nonlinear regression models are widely used in analysis of statistical data [14, 16]. In many applications the observed data are derived from a mixture of components with different dependencies between the variables in different components. In this case a finite mixture model can be used to describe the data [15, 19, 2]. If the concentrations of components in the mixture are different for different observations then the model of mixture with varying concentrations (MVC) can be applied [1, 12, 11]. Parametric models of nonlinear regression mixtures were considered in [5, 4]. Estimation in linear regression MVC models was studied also in [6, 8].

In this paper we adopt a semiparametric approach with the use of modified least squares (mLS) technique. The consistency of mLS estimators in regression MVC models was demonstrated in [9]. Our aim is to derive conditions of mLS estimators asymptotic normality and construct confidence sets for the true values of parameters.

The rest of the paper is organized as follows. In Section 2 we introduce the regression mixture model and the mLS estimator for the regression parameters. Asymptotic behavior of the estimator is discussed in Section 3. Confidence ellipsoids for the parameters are constructed in Section 4. Results of simulations are presented in Section 5. Conclusive remarks are made in Section 6.

In this paper we consider regression technique application to data, which are described by the model of mixture with varying concentrations. It means that each observed subject belongs to one of M different sub-populations (i.e. components of the mixture). We observe n such subjects O1 , …, On . The true number of component which Oj belongs to will be denoted by κj . These numbers are not observed, but one knows the probabilities pj;nk=P{κj=k}. These probabilities are called the mixing probabilities or concentrations of the components at j-th observation.

For each subject Oj , one observes a set of numerical variables ξj;n=ξj=(Yj,Xj1,…,Xjm) , where Y is the response and Xj=(Xj1,…,Xjm)T is the vector of independent variables in the regression model Y=g(X;ϑ)+ε, where g is a known regression function, ϑ=(ϑ1,…,ϑd)T is a vector of unknown regression coefficients, ε is an unobservable regression error. In fact, the coefficients of the model can be different for different components: ϑ=ϑ(k) if κj=k . The distribution of ε can also depend on κj . These dependencies are described in the following model (1) Yj=g(Xj,ϑ(κj))+εj(κj). Here ϑ(k)∈Θ(k)⊆Rd is the vector of unknown regression coefficients corresponding to the k-th mixture component, εj(k) , j=1,…,n , k=1,…,M are independent random variables with distribution dependent on k but not on j.

We will assume that Eεj(k)=0,Varεj(k)=σ2(k)<∞. (The values of σ2(k) are unknown.) The independent variables vectors Xj are considered as random vectors with distribution possibly dependent on κj . It is assumed that the error term εj(κj) and Xj are conditionally independent for given κj . The vectors (ξj,κj) are independent for different j.

In what follows we will frequently use expectations and probabilities connected with different mixture components. To present them in a compact form, we introduce formal random vectors (Y(m),X(m),ε(m)) which have conditional distribution of (Yj,Xj,εj(κj)) given κj=m , i.e., the distribution of the m-th component.

We will also denote by p;n the matrix of all concentrations for all observations and all components: p;n=p1;n1…p1;nM⋮⋱⋮pn;n1…pn;nM, pj;n=(pj;n1,…,pj;nM)T , p;nm=(p1;nm,…,pn;nm)T .

Similar notation is used for the weights matrix a;n introduced below.

We are interested in estimating the parameters ϑ(k) for different components. The considered estimators are based on the modified least squares (mLS) approach. Namely, we consider the weighted least squares functional J(k)(t)= ∑j=1naj;nk(Yj−g(Xj;t))2, where t∈Θ(k) is a formal parameter, aj;nk are some weights aimed to single out the k-th mixture component and suppress influence of all other components on the functional J(k) . In this presentation, we restrict ourselves by the minimax weights matrix defined as (2) a;n=Γ;n−1p;n, where Γ;n=p;nTp;n. (It is assumed here that Γ;n is nonsingular. See [11, 12] for the minimax properties of these weights.) It is readily seen that (3) (a;nk)Tp;nm=1{m=k}. (Here 1{A} is the indicator function of an event A.) So EJ(k)(t)= ∑j=1naj;nk ∑m=1Mpj;nmE(Y(m)−g(X(m),t))2=E(Y(k)−g(X(k),t))2=¯E(g(X(k),ϑ(k))−g(X(k),t))2+σ2(k)=defJ¯(t). The minimum of J¯(t) is attained at t=ϑ(k) . Thus, if ϑ(k) is the unique minimum point, one expects that under suitable conditions J(k)(t)→J¯(t) by the law of large numbers, and argmint∈Θ(k)J(t)→ϑ(k) as n→∞ . If g is smooth enough, the argmin can be found as a solution to J˙(k)(t)=0, where J˙(k)(t) denotes the vector of partial derivatives of J(k)(t) by each entry of t.

In what follows we define the mLS estimator ϑˆ;n(k) for ϑ(k) as a statistic which is a solution to (4) J˙(k)(t)= ∑j=1naj;nk(Yj−g(Xj,t))g˙(Xj,t)=0, where g˙(Xj,t)=∂g(Xj,t)∂t1,…,∂g(Xj,t)∂tdT. If there are many solutions to (4) then ϑˆ;n(k) can be taken any of them, but it must be a measurable function from the observed data (Yj,Xj) , j=1,…,n .

Note that so defined mLS estimator can be a point of local minimum of J(k)(t) . But we still call it mLS since in was obtained by a modification of the LS technique.

In this section, we consider asymptotic behavior of ϑˆ;n(k) as the sample size n tends to infinity. Let us start with some general assumptions on the model.

In this paper we make no assumptions on connections between p;n and p;m , when n≠m and don’t assume that they tend to some limit as n→∞ . Some assumptions are made only on asymptotic behavior of some averaged characteristics of concentrations.

Note that if a significant fraction of pj;nk is bounded away from zero, then entries of the matrix Γ;n=p;nTp;n are of order n as n→∞ . In what follows we will assume that the limit matrix (5) limn→∞1nΓ;n=Γ exists and is nonsingular.

Then the weights aj;nk are of order 1/n as n→∞ and the sums of the form ∑j=1naj;nkaj;nmpj;nlpj;ni are of order 1/n as well.

We will assume that the limits (6) limn→∞n ∑j=1naj;nkaj;nmpj;nlpj;ni=def⟨akamplpi⟩ exist, for all k, m, l, i=1,…,M . Then limn→∞n ∑j=1naj;nkaj;nmpj;nl=def⟨akampl⟩= ∑i=1M⟨akamplpi⟩, since ∑i=1Mpj;ni=1 .

We will also denote (7) h(ξj,t)=(Yj−g(Xj,t))g˙(Xj,t), so ϑˆ;n(k) is a solution to (8) ∑j=1naj;nkh(ξj,t)=0. Note that (8) is an unbiased generalized estimating equation (GEE, see [17], Section 5.4).

Conditions of the ϑˆ;n(k) consistency are presented in the next statement.

Now consider the asymptotic normality of ϑˆ;n(k) . We will start with a result formulated in more general terms of GEE estimation.

Assume that Ξ;n=(ξj;n , j=1,…,n) are random observations in a measurable space X described by the model of mixture with varying concentrations (MVC), i.e., (9) P{ξj;n∈A}= ∑m=1Mpj;nmF(m)(A), where F(m)(A)=P{ξ(m)∈A} is the distribution of the observed variable ξ for subjects from the m-th mixture component. Let ϑ=ϑ(F)∈Rd be a functional on a set of possible components’ distributions. To estimate ϑ(k)=ϑ(F(k)) , we consider an estimating equation of the form (8), where h=(h1,…,hd)T is some estimating function h:X×Rd→Rd such that Eh(ξ(k),ϑ(k))=0. (I.e., h is an unbiased estimating function.) Any statistic ϑˆ;n(k) is called a GEE-estimator for ϑ(k) if it is an a.s. solution to (8), i.e., (10) H(k)(ϑˆ;n(k))=def ∑j=1naj;nkh(ξj,ϑˆ;n(k))=0a.s. We consider the joint parameters vector ϑ=((ϑ(1))T,…,(ϑ(M))T)T and the corresponding estimator ϑˆ;n=((ϑˆ;n(1))T,…,(ϑˆ;n(M))T)T and derive conditions, under which (11) n(ϑˆ;n−ϑ)⟶WN(0,S), where S is some matrix which we now will describe. Denote (12) V(m)=defEh˙(ξ(m),ϑ(m)), where h˙(ξ(m),t)=∂h1(ξ(m),ϑ(m))∂t1…∂h1(ξ(m),ϑ(m))∂td⋮⋱⋮∂hd(ξ(m),ϑ(m))∂t1…∂hd(ξ(m),ϑ(m))∂td. Then Z(m,l)= ∑i=1M⟨amalpi⟩Eh(ξ(i),ϑ(m))hT(ξ(i),ϑ(l))− ∑i=1M ∑k=1M⟨amalpipk⟩Eh(ξ(i),ϑ(m))EhT(ξ(k),ϑ(l)),S=S(1,1)…S(1,M)⋮⋱⋮S(M,1)…S(M,M), where (13) S(m,l)=(V(m))−1Z(m,l)(V(m))−T. (Here and below V−T=def(V−1)T .)

Return to the regression mixture model (1). Obviously it is a partial case of the MVC model (9). How the matrices V(m) and Z(m,l) can be represented for the regression mixture model?

Assume that h is defined by (7) and the function g(x,t) has second derivatives by t: g¨(Xj,t)=∂2g(Xj,t)∂t1∂t1…∂2g(Xj,t)∂t1∂td⋮⋱⋮∂2g(Xj,t)∂td∂t1…∂2g(Xj,t)∂td∂td. Then V(m)=E∂∂t[(g(X(m),ϑ(m))+ε(m)−g(X(m),t))g˙(X(m),t)]t=ϑ(m)=−Eg˙(X(m),ϑ(m))(g˙(X(m),ϑ(m)))T+Eε(m)g¨(X(m),ϑ(m)). The second term is zero since ε(m) is independent from X(m) and Eε(m)=0 . So (15) V(m)=−Eg˙(X(m),ϑ(m))(g˙(X(m),ϑ(m)))T. A similar algebra yields Z(m,k)= ∑l=1M⟨amakpl⟩[σ2(l)A(l,k,m)+B(l,k,m)]− ∑i,l=1M⟨amakplpi⟩(G(i,i,k)−G(i,k,k))(G(l,l,m)−G(l,m,m)), where A(l,k,m)=Eg˙(X(l),ϑ(k))g˙T(X(l),ϑ(m)),B(l,k,m)=E(g(X(l),ϑ(l))−g(X(l),ϑ(k)))×(g(X(l),ϑ(l))−g(X(l),ϑ(m)))g˙(X(l),ϑ(k))g˙T(X(l),ϑ(m)),G(l,m,k)=Eg(X(l),ϑ(m))g˙(X(l),ϑ(k)).

Apply the results of Section 3 to the construction of asymptotic confidence sets for ϑ(k) . For any t∈Rd and any nonsingular S∈Rd×d , define T(k)(t,S)=n(ϑˆ;n(k)−t)TS−1(ϑˆ;n(k)−t).

It is obvious that if Theorem 2 holds and S(k,k) is nonsingular, then (16) T(k)(ϑ(k),S(k,k))⟶Wχd2, where χd2 is the χ2 -distribution with d degrees of freedom. Note that (16) holds also if S(k,k) is replaced by a consistent estimator Sˆ;n(k,k) . Let χα be the α-upper quantile of χd2 . Then the set B;nk(α)={t∈Rd:T(k)(ϑ(k),Sˆ(k,k))<χα} is an asymptotic α-level confidence set for ϑ(k) in the sense that P{ϑ(k)∈B;nk(α)}→1−α as n→∞ .

To accomplish the confidence set construction, we need convenient conditions for the S(k,k) nonsingularity and a consistent estimator of this matrix.