The convergence of the scaled cumulant generating functions of a sequence of random variables implies a large deviation principle; this is known as the Gärtner–Ellis condition [6, p. 43]. Our main result is that condition for the square of an asymptotically stationary Gaussian process. Reasons for studying squared Gaussian processes come from different fields: large deviation theory [19, 5], time series analysis [10], or ancestry-dependent branching processes [16]. Since only nonnegative real-valued random variables are considered here, we shall use logarithms of Laplace transforms instead of cumulant generating functions.

Theorem 1 yields as a particular case the following result of weak multiplicative ergodicity.

The analogue for finite-state Markov chains has long been known [6, p. 72]. It was extended to strong multiplicative ergodicity of exponentially converging Markov chains by Meyn and his coworkers; see [14]. In [13], the square of a Gauss–Markov process was studied, strong multiplicative ergodicity was proved, and the limit was explicitly computed. This motivated the present generalization.

The particular case of a centered stationary process ( m(t)=0 , K(t,s)=k(t−s) ) can be considered as classical: in that case, the limit (4) follows from Szegő’s theorem on Toeplitz matrices: see [9, 4] as a general reference on Toeplitz matrices and [2] for a review of probabilistic applications of Szegő’s theory. The extension to the centered asymptotically stationary case follows from the notion of asymptotically equivalent matrices in the L2 sense; see Section 7.4, p. 104, of [9], and [8]. The noncentered stationary case ( m(t)=m∞ and K(s,t)=k(s−t) ) has received much less attention. In Proposition 2.2 of [5], the large deviation principle is obtained for a squared noncentered stationary Gaussian process. There, the centered case is deduced from Szegő’s theorem, whereas the noncentered case follows from the contraction principle. A similar approach to the general case can be found in [1].

We propose here a different method. Instead of the spectral decomposition and Szegő’s theorem, a Wiener–Hopf factorization is used. The limits (4) and (5) are both deduced from the asymptotics of that factorization. The technique is close to those developed in [12] and used in [13]. One advantage is that the coefficients of the Wiener–Hopf factorization can be given a probabilistic interpretation in terms of a regression problem. This approach will be detailed in Section 2.

To go from the stationary to the asymptotically stationary case, the asymptotic equivalence of matrices is needed. But the classical L2 definition of [8, Sect. 2.3] does not suffice for the noncentered case. A stronger notion, linked to the L1 norm of vectors instead of the L2 norm, will be developed in Section 3.

Joining the stationary case to asymptotic equivalence, we get the conclusion of Theorem 1, but only for small enough values of α. To deduce that the convergence holds for all α⩾0 , an extension of Lévy’s continuity theorem will be used: if both (Lt(α))1/t and e−ℓ(α) are the Laplace transforms of probability distributions on R+ , then the convergence over an interval implies the weak convergence of measures and hence the convergence of Laplace transforms for all α⩾0 . In fact, (Lt(α))1/t and e−ℓ(α) both are the Laplace transforms of infinitely divisible distributions, more precisely, convolutions of gamma distributions with Poisson compounds of exponentials. Details will be given in Section 4, together with the particular case of a Gauss–Markov process.

This section treats the stationary case: m(t)=m∞ and K(s,t)=k(t−s) . We shall denote by ct=(m∞)s=0,…,t−1 the constant vector with coordinates all equal to m∞ and by Ht the Toeplitz matrix with symbol k: Ht=(k(s−r))s,r=0,…,t−1 . The main result of this section is a particular case of Theorem 1. It entails Proposition 2.2 of Bryc and Dembo [5].

Denote by mt and Kt the mean and covariance matrix of the vector (Xs)s=0,…,t−1 . The Laplace transform of the squared norm of a Gaussian vector has a well-known explicit expression; see, for instance, [19, p. 6]. The identity matrix indexed by 0,…,t−1 is denoted by It , and the transpose of a vector m is denoted by m∗ . Then (7) Lt(α)=(det(It+2αKt))−1/2exp(−αmt∗(It+2αKt)−1mt), In the stationary case, mt=ct and Kt=Ht . From (7) we must prove that the following two limits hold: (8) limt→+∞12tlog(det(It+2αHt))=ℓ0(α)=14π∫02πlog(1+2αf(λ))dλ and (9) limt→+∞αtct∗(It+2αHt)−1ct=ℓ1(α)=m∞2α(1+2αf(0))−1. Here, It+2αHt will be interpreted as the covariance matrix of the random vector (Ys)s=0,…,t−1 from the process (10) Y=ε+2αZ, where ε=(εt)t∈Z is a sequence of i.i.d. standard normal random variables, independent from Z. The limits (8) and (9) will be deduced from a Cholesky decomposition of It+2αHt . We begin with an arbitrary positive definite matrix A. The Cholesky decomposition writes it as the product of a lower triangular matrix by its transpose. Thus, A−1 is the product of an upper triangular matrix by its transpose. Write it as A−1=T∗DT , where T is a unit lower triangular matrix (the diagonal coefficients equal to 1), and D is a diagonal matrix with positive coefficients. Denote by G the lower triangular matrix  DT . Then GA=(T∗)−1 is a unit upper triangular matrix. Hence, the coefficients G(s,r) of G are uniquely determined by the following system of linear equations: for 0⩽s⩽t , (11) ∑r=0tG(t,r)A(r,s)=δt,s, where δt,s denotes the Kronecker symbol equal to 1 if t=s and 0 else. Notice that A−1=G∗D−1G , and TAT∗=D−1 , where D is the diagonal matrix with diagonal entries G(s,s) . In particular, (12) det(A)=(∏sG(s,s))−1, and for any vector m=(m(r)) , (13) m∗A−1m=∑s1G(s,s)( ∑r=0sG(s,r)m(r))2. Here is the probabilistic interpretation of the coefficients G(t,s) . Consider a centered Gaussian vector Y with covariance matrix A. For t=0,…,n , denote by Y⟦0,t⟧ the σ-algebra generated by Y0,…,Yt , and by νt the partial innovation νt=Yt−E[Yt|Y⟦0,t−1⟧] with the convention ν0=Y0 . Using elementary properties of Gaussian vectors, it is easy to check that (14) νt=1G(t,t) ∑r=0tG(t,r)Yr. Moreover, the νt are independent, and the variance of νt is 1/G(t,t) .

When this is applied to A=It+2αHt , another interesting interpretation arises. For t=0,…,n , (G(t,s))s=0,…,t is the unique solution to the system (15) G(t,s)+2α ∑r=0tG(t,r)k(r−s)=δt,s. Observe that Eqs. (15) are the normal equations of the regression of the εt over the Yt in the model (10). Actually, since E[YrYs]=δs,r+2αk(r−s) and E[εtYs]=δt,s , setting (16) μt=G(t,t)νt= ∑r=0tG(t,r)Yr, Eq. (15) says that for s=0,…,t , E[μtYs]=E[εtYs]. This means that μt=E[εt|Y⟦0,t⟧]. Obviously, the μt are independent, the variance of μt is G(t,t) , and the filtering error is E[(εt−μt)2]=1−G(t,t). In particular, it follows that 0<G(t,t)<1 .

The asymptotics of G(t,s) will now be related to the spectral density f. Denote gt(s)=G(t,t−s) . A change of index in (15) shows that (gt(s))s=0,…,t is the unique solution to the system (17) gt(s)+2α ∑r=0tgt(r)k(s−r)=δs,0.

The proof is equivalent to writing the Wiener–Hopf factorization of the operator I+2αH : compare with Section 1.5 of [4], in particular, with the proof of Theorem 1.14 on p. 17. The main idea is to reduce Eq. (18) to the problem of finding a sectionally holomorphic function satisfying a boundary condition on the unit circle. This idea is originally due to Krein [15].

Here is the probabilistic interpretation. Consider a centered stationary process (Yt)t∈Z with covariance function A(t,s)=a(t−s) . For s⩽t , denote by Y⟦s,t⟧ the σ-algebra generated by (Yr)r=s,…,t . Consider again the partial innovation νt=Yt−E[Yt|Y⟦0,t−1⟧] . From (14) and using stationarity, νt has the same distribution as ηt=1G(t,t) ∑r=0tG(t,t−r)Y−r, which is ηt=Y0−E[Y0|Y⟦−t,−1⟧]. As t tends to infinity, ηt converges almost surely to η∞=Y0−E[Y0|Y⟦−∞,−1⟧]. Observe by stationarity that for all r, η∞=DYr−E[Yr|Y⟦−∞,r−1⟧], which is the innovation process associated to Y. Now the variance of νt , 1/G(t,t) , tends to the variance of η∞ . By the Szegő–Kolmogorov formula (see, e.g., Theorem 3 on p. 137 of [10]) that variance is exp(12π∫02πlog(ϕ(λ))dλ), where ϕ(λ) is the spectral density of Y. Let X be a centered stationary process with covariance function k, ε be a standard Gaussian noise, and Y=ε+2αX . The spectral densities ϕ of Y and f of X are related by ϕ(λ)=1+2αf(λ) . Hence, limt→+∞var(νt)=limt→+∞1G(t,t)=exp(12π∫02πlog(1+2αf(λ))dλ), which is equivalent to (19).

Alternatively, observe that, due to stationarity, μt defined by (16) has the same distribution as ξt= ∑r=0tG(t,t−r)Y−r, which is ξt=E[ε0|Y⟦−t,0⟧]. As t tends to infinity, ξt converges a.s. to ξ∞=E[ε0|Y⟦−∞,0⟧]. Of course, since E[ε−sY−r]=δs,r for all s=0,…,t , E[ξtε−s]=G(t,t−s). Hence, the limiting property (21) says that E[ξ∞ε−s]=limt→+∞G(t,t−s)=g(s). In fact, ξ∞ admits the representation ξ∞= ∑s=0+∞g(s)Y−s. Similarly, for all t, E[εt|Y⟦−∞,t⟧]= ∑s=0+∞g(s)Yt−s, which means that (g(s)) realizes the optimal causal Wiener filter of εt from the Yt−s .

Now, Proposition 2 is a straightforward consequence of Proposition 3. Proof.

Let the coefficients gτ(s) be defined by (17). Applying (12) to A=It+2αHt , we get (det(It+2αHt))−1/2=( ∏τ=0t−1gτ(0))1/2. Therefore, 1tlog((det(It+2αHt))−1/2)=12t ∑τ=0t−1log(gτ(0)). From Proposition 3 we have limτ→+∞gτ(0)=g(0)=exp(−12π∫02πlog(1+2αf(λ))dλ). Hence, limt→+∞1tlog((det(It+2αHt))−1/2)=−ℓ0(α). Applying now (13) to A=It+2αHt , we get ct∗Gt∗Dt−1Gtct=m∞2 ∑τ=0t−11gτ(0)( ∑s=0τgτ(s))2. From Proposition 3 we have limτ→+∞1gτ(0)( ∑s=0τgτ(s))2=1g(0)( ∑s=0+∞g(s))2=(1+2αf(0))−1. Hence, limt→+∞αtct∗(It+2αHt)−1ct=ℓ1(α).  □

Proposition 2 only treats the stationary case. To extend the result under the hypotheses of Theorem 1, a notion of asymptotic equivalence of matrices and vectors is needed. It is developed in this section.

From (7), we must prove that, under the hypotheses of Theorem 1, (27) limt→+∞12tlog(det(It+2αKt))=ℓ0(α)=14π∫02πlog(1+2αf(λ))dλ and (28) limt→+∞αtmt∗(It+2αKt)−1mt=ℓ1(α)=m∞2α(1+2αf(0))−1. If Kt=Ht (centered stationary case), then (27) is (8). It can also be obtained by a straightforward application of Szegő’s theorem; see [4, 2]. Relation (27) (centered asymptotically stationary case) is a consequence of the theory of asymptotically Toeplitz matrices; see Section 7.4 on p. 104 of [9] and also [8, Theorem 4 on p. 178]. Asymptotic equivalence of matrices in Szegő’s theory is taken in the L2 sense, which is weaker than that considered here. In other words, (27) holds under weaker hypotheses than (H1–H5). In order to prove (28), we shall develop asymptotic equivalence of matrices and vectors along the same lines as [8, Sect. 2.3], but in a stronger sense, replacing L2 by L∞ and L1 , for boundedness and convergence. The norms used here for a vector v=(v(s))s=0,…,t−1 are ‖v‖∞= maxs=0t−1|v(s)|and‖v‖1= ∑s=0t−1|v(s)|. For symmetric matrices, the norm subordinate to ‖·‖∞ is equal to the norm subordinate to ‖·‖1 . It will be denoted by ‖·‖ and referred to as the strong norm. For A=(A(s,r))s,r=0,…,t−1 such that A∗=A , ‖A‖= maxs=0t−1 ∑r=0t−1|A(s,r)|=max‖v‖∞=1‖Av‖∞= maxr=0t−1 ∑s=0t−1|A(s,r)|=max‖v‖1=1‖Av‖1. The following weak norm will be denoted by |A| : |A|=1t ∑s,r=0t−1|A(s,r)|. Clearly, |A|⩽‖A‖ . Moreover, the following bounds hold.

Here is a definition of asymptotic equivalence for vectors.

Hypotheses (H1) and (H4) imply that mt∼ct .

Asymptotic equivalence for matrices is defined as follows (compare with [8, p. 172]).

Here are some elementary results, analogous to those stated in Theorem 1 on p. 172 of [8].