Let $ f $ be a decreasing density with support $ [0, \infty) $. Denote by $ F_n $ the empirical distribution function of a sample $ X_1, ..., X_n $ from $ f $. Let $ \hat{F}_n $ be the concave majorant of $ F_n $ on $ [0, \infty) $, i.e. the smallest concave function such that

$ \hat{F}_n(t)\geq F_n(t), \quad \hat{F}_n(0) = 0, \quad t\in[0, \infty). $

The Grenander estimator $ \hat{f}_n $ is defined as the left derivative of $ \hat{F}_n $.

Prakasa Rao [9] obtained the earliest result on the asymptotic pointwise behavior of $ \hat{f}_n(t) $ with $ t\in(0, \infty) $

$ \left|4f(t)f'(t)\right|^{-1/3}n^{1/3}\left(\hat{f}_n(t)-f(t)\right)\stackrel{d}{\longrightarrow} \rm{argmax}_{t\in\mathbb{R}}\left\{W(t)-t^2\right\}, $

where $ W $ denotes a two-sided standard Wiener process originating from zero, $ \stackrel{d}{\longrightarrow} $ means convergence in distribution. Then Groeneboom [6] provided an elegant proof of the same result based on inverse process of $ \hat{f}_n $, which has become a cornerstone in this field. For the associated moderate deviations, one can see Gao et al. [4].

It should be noted that the Grenander estimator $ \hat{f}_n $ is not consistent at boundaries ([1], [10]). This phenomenon has great influences on the global measures of deviation, such as the $ L_k $-distances with $ k>1 $ ([2], [7]) and $ L_{\infty} $-distance ([3]), because the inconsistency at the boundaries will dominate the convergence.

To make the properties of $ \hat{f}_n $ near boundaries more clear, Kulikov and Lopuhaä [8] considered asymptotic distribution of

$ n^{\alpha}\left(\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha})\right), $

where $ c, \alpha>0, 0<\alpha<1 $. To be explicit, for the left boundary zero, suppose the following conditions hold:

(C1) $ 0<f(0) = \lim_{t\downarrow0}f(t)<\infty $;

(C2) there exists some positive constant $ \varepsilon_0 $ such that $ f $ has $ k $-th order continuous derivative in $ (0, \varepsilon_0] $ and $ f(\varepsilon_0)\neq0 $. Moreover $ 0<|f^{(k)}(0)|\leq\sup_{t\geq0}|f^{(k)}(t)|<\infty $, with $ f^{(k)}(0) = \lim_{t\downarrow0}f^{(k)}(t) $ and $ f^{(i)}(0) = 0 $ for $ 1\leq i\leq k-1 $.

Kulikov and Lopuhaä ([7]) formulated that, as $ 0<\alpha<1/(2k+1) $,

$ c^{(1-k)/3}2^{-2/3}((k-1)!)^{1/3}n^{1/3+\alpha(k-1)/3}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha})}{\big|f(0)f^{(k)}(0)\big|^{1/3}} \stackrel{d}{\longrightarrow} \rm{argmax}_{t\in\mathbb{R}}\left\{W(t)-t^2\right\}. $

Moreover, in the case of $ 1/(2k+1)<\alpha<1 $,

$ n^{(1-\alpha)/2}c^{1/2}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha})}{f^{1/2}(0)}\stackrel{d}{\longrightarrow} \sqrt{\rm{argmax}_{t\in[0, \infty)}\left\{W(t)-t\right\}}. $

In this paper, the moderate deviations of $ \hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha}) $ in the above two cases ($ 0<\alpha<1/(2k+1) $ and $ 1/(2k+1)<\alpha<1 $) will be considered. By using strong approximation technique and comparison method, we obtain the following main results.

Theorem 1.1 When $ 0<\alpha<1/(2k+1) $, let $ \ell_n $ satisfy as $ n\to\infty $

$ \ell_n\to\infty, \quad \frac{n^{\alpha}}{\ell_n^{15}(\log n)^2}\to\infty. $

Then, under conditions (C1) and (C2), the sequence

$ \left\{\frac{n^{1/3+\alpha(k-1)/3}}{\ell_n}c^{(1-k)/3}2^{-2/3}((k-1)!)^{1/3}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha})} {\big|{f}(0){f}^{(k)}(0)\big|^{1/3}}, n\geq 1\right\} $

satisfies the moderate deviations in $ \mathbb R $ with speed $ \ell_n^3 $ and rate function $ I(x) = \frac{2}{3}|x|^3, $ that is, for any open subset $ G $ of $ \mathbb R $,

$ \liminf\limits_{n\rightarrow \infty}\frac 1{\ell_n^3} \log {P}\left(\frac{n^{1/3+\alpha(k-1)/3}}{\ell_n}c^{(1-k)/3}2^{-2/3}((k-1)!)^{1/3}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha})} {\big|{f}(0){f}^{(k)}(0)\big|^{1/3}}\in G\right) \geq - \inf\limits_{x \in G}I(x), $

and for any closed subset $ F $ of $ \mathbb R $,

$ \limsup\limits_{n\rightarrow \infty} \frac 1{\ell_n^3} \log {P}\left(\frac{n^{1/3+\alpha(k-1)/3}}{\ell_n}c^{(1-k)/3}2^{-2/3}((k-1)!)^{1/3}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha})} {\big|{f}(0){f}^{(k)}(0)\big|^{1/3}}\in F\right) \leq -\inf\limits_{x\in F} I(x). $

Remark 1 For any $ x>0 $, by Theorem 1.1, we have

$ \lim\limits_{n\rightarrow \infty} \frac 1{\ell_n^3} \log {P}\left(\pm\frac{n^{1/3+\alpha(k-1)/3}}{\ell_n}c^{(1-k)/3}2^{-2/3}((k-1)!)^{1/3}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha})} {\big|{f}(0){f}^{(k)}(0)\big|^{1/3}}\geq x\right) = -\frac{2}{3}|x|^3. $

Theorem 1.2 When $ 1/(2k+1)<\alpha<1 $, let $ \{\lambda_n\} $ satisfy that

$ \lambda_n\to\infty, \quad \frac{n^{(1-\alpha)}}{\lambda_n^{30}(\log n)^6}\to\infty, \quad \frac{n^{(2k+1)\alpha-1}}{\lambda_n^{4k+10}}\to \infty. $

Then, under conditions (C1) and (C2), the sequence

$ \left\{\frac{n^{(1-\alpha)/2}}{\lambda_n}c^{1/2}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha}t)}{{f}^{1/2}(0)}, n\geq 1\right\} $

satisfies the moderate deviations in $ \mathbb R^+ $ with speed $ \lambda_n^2 $ and rate function $ J(x) = \frac{x^2}{2}. $

Remark 2 For any $ x>0 $, by Theorem 1.2, we have

$ \lim\limits_{n\rightarrow \infty} \frac 1{\lambda_n^2} \log {P}\left(\frac{n^{(1-\alpha)/2}}{\lambda_n}c^{1/2}\frac{\hat{f}_n(cn^{-\alpha})-f(cn^{-\alpha}t)}{{f}^{1/2}(0)}\geq x\right) = -\frac{x^2}{2}. $

Remark 3 If $ f $ has compact support, without loss of generality, assume it is the interval $ [0, 1] $. The moderate deviations of $ \hat{f}_n $ near the right boundary $ 1 $ (similar to Theorem 1.1 and Theorem 1.2) can also be obtained, and the details are omitted here.

For a detail study of the moderate deviations for Grenander estimator near boundaries, please refer to [5].