In this paper, we will first consider the following system

$ \begin{equation} \begin{cases} -\Delta_{\mathbb{B}}u+\lambda_{1}u=\mu_{1}u^{2p-1}+\beta u^{p-1}v^{p}, &x \in {\rm int}~\mathbb{B}, \\ -\Delta_{\mathbb{B}}v+\lambda_{2}v=\mu_{2}v^{2p-1}+\beta v^{p-1}u^{p}, &x \in{\rm int}~ \mathbb{B}, \\ u\geq0, v\geq0 ~~ &{\rm in} ~~ {\rm int}~\mathbb{B}, u=v=0 ~~{\rm on}~~ \partial\mathbb{B}, \end{cases} \end{equation} $

(1.1)

where $N\geq5$, $p=\frac{N}{N-2}, 2^{\ast}=\frac{2N}{N-2}$, $-\lambda_{1}(\mathbb{B})<\lambda_{1}, \lambda_{2}<0, \mu_{1}, \mu_{2}>0$ and $\beta\neq0$. Here $\mathbb{B}$ is $ [0, 1)\times X$ and $X\subset\mathbb{R}^{N-1}$ is a smooth compact domain, $\lambda_{1}(\mathbb{B})$ is the first eigenvalue of $-\Delta_\mathbb{B}$ with zero Dirichlet condition on $\partial\mathbb{B}$, $ \Delta_{\mathbb{B}}$ =$(x_{1}\partial_{x_{1}})^{2}+\partial^{2}_{x_{2}}+\cdots+\partial^{2}_ {x_{N}}$. We will look for the positive least energy solutions for (1.1) in the cone Sobolev space $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\mathbb{B})$, which was introduced in [13]. In [2], Chen-Liu-Wei considered the following problem

$ \begin{equation} \begin{cases} -\Delta_{\mathbb{B}}u+\lambda u=| u|^{2^{*}-2}u, &u\in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\mathbb{B}), \\ u=0~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~&\mbox{on}~~\partial\mathbb{B}, \end{cases} \end{equation} $

(1.2)

and got a positive solution $\varphi $. Recently, the authors in [8] also studied the positive least energy solutions for $p$-Laplacian system. Our study is in fact motivated by the study of Chen-Zou (see [1]), and we investigate the semi-linear equations with critical cone Sobolev exponent terms.

We call a solution $(u, v)\in H$ nontrivial if $u\not\equiv0, v\not\equiv0$, where $H:=\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\mathbb{B})\times \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\mathbb{B})$. The weak solutions of (1.1) are the critical points of the functional $J :H\rightarrow\mathbb{R}$, which is given by

$ \begin{align*} J(u, v)=&\frac{1}{2}\int_{\mathbb{B}}(| \nabla_{\mathbb{B}}u|^{2}+\lambda_{1}u^{2})\frac{dx_{1}}{x_{1}}dx^{'}+\frac{1}{2}\int_{\mathbb{B}}(| \nabla_{\mathbb{B}}v|^{2}+\lambda_{2}v^{2})\frac{dx_{1}}{x_{1}}dx^{'} \\ &-\frac{1}{2p}\int_{\mathbb{B}}(\mu_{1}| u|^{2p}+2\beta | u|^{p}| v|^{p}+\mu_{2}|v|^{2p})\frac{dx_{1}}{x_{1}}dx^{'}. \end{align*} $

(1.3)

We say that a solution $(u, v)$ of (1.1) is a least energy solution if $(u, v)$ is nontrivial and $J(u, v)\leq J(\varphi, \psi)$ for any other nontrivial solution $(\varphi, \psi)$ of (1.1). If we define a " Nehari" manifold (see [1, 4-7, 9])

$ \begin{equation*} \mathcal{N}=\{(u, v)\in H:u\not\equiv0, v\not\equiv0, J'(u, v)(u, 0)=J'(u, v)(0, v)=0\}, \end{equation*} $

then any nontrivial solutions of (1.1) belong to $\mathcal{N}$, here $J'(\cdot, \cdot)$ is the Fr${\rm \acute{e}}$chet differentiation of $J$. We define the least energy of (1.3) as

$ \begin{equation*} A:=\inf\limits_{(u, v)\in\mathcal{N}}J(u, v)=\inf\limits_{(u, v)\in\mathcal{N}}\{\frac{1}{N}\int_{\mathbb{B}}(| \nabla_{\mathbb{B}}u|^{2}+\lambda_{1}u^{2}+| \nabla_{\mathbb{B}}v|^{2}+\lambda_{2}v^{2})\frac{dx_{1}}{x_{1}}dx^{'}\}. \end{equation*} $

If the equation

$ \begin{cases} \mu_{1}d^{p-1}+\beta d^{\frac{p}{2}-1}g^{\frac{p}{2}}=1, \\ \beta d^{\frac{p}{2}}g^{\frac{p}{2}-1}+\mu_{2}g^{p-1}=1, \\ d>0, g>0 \end{cases} $

(1.4)

has a solution $(d_{0}, g_{0})$ with

$ \begin{equation*} d_{0}=\min\{d:(d, g)~~{\rm satisfies}~~ (1.4)\} , \end{equation*} $

(1.5)

then we prove the following theorem.

Theorem 1.1  Let $(d_{0}, g_{0})$ be a solution of (1.4) with $ d_{0}$ in (1.5) and $-\lambda_{1}(\mathbb{B})<\lambda_{1}=\lambda_{2}=\lambda<0$. Then for any $\beta>0$, $(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)$ is a positive solution of (1.1). Moreover, if $\beta\geq\frac{2}{N-2}\max\{\mu_{1}, \mu_{2}\}$, then we have $J(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)=A$, that is, $(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)$ is a positive least energy solution of (1.1).

In the second part of this paper, we consider the existence of the least energy solution of the following problem

$ \begin{cases} -\Delta_{\mathbb{B}}u=\mu_{1}| u|^{2p-2}u+\beta | u|^{p-2}u|v|^{p}, x \in \mathbb{R}_{+}^{N}, \\ -\Delta_{\mathbb{B}}v=\mu_{2}|v|^{2p-2}v+\beta| v|^{p-2}v | u|^{p}, x \in \mathbb{R}_{+}^{N}, \\ u, v \in D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N}), \end{cases} $

(1.6)

where $\mathbb{R}_{+}^{N}=\mathbb{R}_{+}\times\mathbb{R}^{N-1}$ and $D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N})=: \{u\in L_{2^{*}}^{\frac{N}{2^{*}}}:|\nabla_{\mathbb{B}}u |\in L_{2}^{\frac{N}{2}}(\mathbb{R}_{+}^{N})\}$ with norm $\| u \|_{D_{2}^{1, \frac{N}{2}}}=(\displaystyle\int_{\mathbb{R}_{+}^{N}}| \nabla_{\mathbb{B}}u|^{2}\frac{dx_{1}}{x_{1}}dx^{'})^{\frac{1}{2}}$. Let $D:=D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N})\times D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N})$ and the energy functional $E$ for (1.6) is defined as

$ \begin{equation*} E(u, v)=\frac{1}{2}\int_{\mathbb{R}_{+}^{N}}(| \nabla_{\mathbb{B}}u|^{2}+| \nabla_{\mathbb{B}}v|^{2})\frac{dx_{1}}{x_{1}}dx^{'} -\frac{1}{2p}\int_{\mathbb{R}_{+}^{N}}(\mu_{1}| u|^{2p}+2\beta | u|^{p}| v|^{p}+\mu_{1}| v|^{2p})\frac{dx_{1}}{x_{1}}dx^{'}. \end{equation*} $

(1.7)

Analogously, we let

$ \begin{eqnarray*} &&\mathcal{M}=\{(u, v)\in D:u\not\equiv 0, v\not\equiv 0, E'(u, v)(u, 0)=E'(u, v)(0, v)=0\}, \\ &&B:=\inf\limits_{(u, v)\in\mathcal{M}}E(u, v)=\inf\limits_{(u, v)\in\mathcal{M}}\{\frac{1}{N}\int_{\mathbb{R}_{+}^{N}}(| \nabla_{\mathbb{B}}u|^{2}+| \nabla_{\mathbb{B}}v|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\}, \end{eqnarray*} $

it is easy to see that any nontrivial solutions of (1.6) belong to $\mathcal{M}$. Then we get the following theorem.

Theorem 1.2  If $\beta<0$, then $B$ is not attained.

(2) If $\beta>0$, then there exists a positive least energy solution $(U, V)$ of (1.6) with $E(U, V)=B$, which is partly radially symmetric decreasing. Furthermore, we have

(2-1) Let $(d_{0}, g_{0})$ be as in Theorem 1.1. If $\beta\geq\frac{2}{N-2}\max\{\mu_{1}, \mu_{2}\}$, then

$ \begin{equation*} E(\sqrt{d_{0}}U_{\varepsilon}, \sqrt{g_{0}}U_{\varepsilon})=B. \end{equation*} $

That is, $(\sqrt{d_{0}}U_{\varepsilon}, \sqrt{g_{0}}U_{\varepsilon})$ is a positive least energy solution of (1.6).

(2-2) There exists $0<\beta_{1}\leq\frac{2}{N-2}\max\{\mu_{1}, \mu_{2}\}$ such that for any $0<\beta<\beta_{1}$, we have a solution $(d(\beta), g(\beta))$ of (1.4) with

$ \begin{equation*} E(\sqrt{d(\beta)}U_{\varepsilon}, \sqrt{g(\beta)}U_{\varepsilon})>B=E(U, V). \end{equation*} $

That is, $(\sqrt{d(\beta)}U_{\varepsilon}, \sqrt{g(\beta)}U_{\varepsilon})$ is a different positive solution of (1.6) with respect to $(U, V)$.

The terminology "partly radially symmetrization decreasing" in Theorem 1.2 will be explained in Section 3. Meanwhile, we will introduce "cone Schwartz symmetrization" in the same section.

The paper is organized as follows. In Section 2, we will give some preliminaries about cone Sobolev spaces and some auxiliary results. In Section 3, we will give the proofs of Theorems 1.1 and 1.2.

Here we first introduce the cone Sobolev spaces. Let $X$ be a closed, compact $C^{\infty}$ manifold of dimension $N-1$, and set $X^{\triangle}=(\overline{\mathbb{R}}_{+}\times X)/(\{0\}\times X)$ which is the local model interpreted as a cone with the base $X$. More details about the manifold with singularities can be found in [10].

Definition 2.1  For $(x_{1}, x^{\prime})\in \mathbb{R}_{+}\times\mathbb{R}^{N-1}$, we say that $u(x_{1}, x^{\prime})\in L_{p}(\mathbb{R}_{+}^{N}, \frac{dx_{1}}{x_{1}}dx^{\prime})$ if

$ \begin{equation*} \|u\|_{L_{p}}=\Big(\int_{\mathbb{R}_{+}}\int_{\mathbb{R}^{N-1}}x_{1}^{N}|u(x_{1}, x^{\prime})|^{p}\frac{dx_{1}}{x_{1}}dx^{\prime}\Big)^{1/p}<+\infty. \end{equation*} $

The weighted $L_{p}$-spaces with weight data $\gamma\in \mathbb{R}$ is denoted by $ L_{p}^{\gamma}(\mathbb{R}_{+}^{N}, \frac{dx_{1}}{x_{1}}dx^{\prime})$, and then $x_{1}^{-\gamma}u(x_{1}, x^{\prime})\in L_{p}(\mathbb{R}_{+}^{N}, \frac{dx_{1}}{x_{1}}dx^{\prime})$ with the norm

$ \begin{equation*} \|u\|_{L_{p}^{\gamma}}=\Big(\int_{\mathbb{R}_{+}}\int_{\mathbb{R}^{N-1}}x_{1}^{N}|x_{1}^{-\gamma}u(x_{1}, x^{\prime})|^{p}\frac{dx_{1}}{x_{1}}dx^{\prime}\Big)^{1/p}<+\infty. \end{equation*} $

Definition 2.2  For $m\in\mathbb{N}$, and $\gamma\in\mathbb{R}$, we define the spaces

$ \begin{equation*} \mathcal{H}_{p}^{m, \gamma}(\mathbb{R}_{+}^{N}):=\Big\{u\in \mathcal{D}^{\prime}(\mathbb{R}_{+}^{N});x_{1}^{\frac{N}{p}-\gamma}(x_{1}\partial_{x_{1}})^{\alpha}\partial_{x^{\prime}}^{\beta}u\in L_{p}(\mathbb{R}_{+}^{N}, \frac{dx_{1}}{x_{1}}dx^{\prime})\Big\} \end{equation*} $

for arbitrary $\alpha\in\mathbb{N}, \beta\in\mathbb{N}^{N-1}$, and $|\alpha|+|\beta|\leq m$. In other words, if $u(x_{1}, x^{\prime})\in \mathcal{H}_{p}^{m, \gamma}(\mathbb{R}_{+}^{N})$, then $(x_{1}\partial_{x_{1}})^{\alpha}\partial_{x^{\prime}}^{\beta}u\in L^{\gamma}_{p}(\mathbb{R}_{+}^{n}, \frac{dx_{1}}{x_{1}}dx^{\prime})$. It's easy to see that $\mathcal{H}_{p}^{m, \gamma}(\mathbb{R}_{+}^{N})$ is a Banach space with the norm

$ \begin{equation*} \|u\|_{\mathcal{H}_{p}^{m, \gamma}(\mathbb{R}_{+}^{N})}=\sum\limits_{|\alpha|+|\beta|\leq m}\Big(\int_{\mathbb{R}_{+}}\int_{\mathbb{R}^{N-1}}x_{1}^{N}|x_{1}^{-\gamma}(x_{1}\partial_{x_{1}})^{\alpha}\partial_{x^{\prime}}^{\beta}u(x_{1}, x^{\prime})|^{p}\frac{dx_{1}}{x_{1}}dx^{\prime}\Big)^{1/p}. \end{equation*} $

We will always denote $\omega(x_{1}, x^{\prime})\in C_{0}^{\infty}(\mathbb{B})$ as a real-valued cut-off function which equals $1$ near $\{0\}\times\partial\mathbb{B}$.

Definition 2.3  Let $\mathbb{B}$ be the stretched manifold to a manifold $B$ with conical singularities. Then $\mathcal{H}_{p}^{m, \gamma}(\mathbb{B})$ for $m\in\mathbb{N}, \gamma\in\mathbb{R}$ denotes the subspace of all $u\in W_{{\rm loc}}^{m, p}({\rm int}~\mathbb{B})$ such that

$ \begin{equation*} \mathcal{H}_{p}^{m, \gamma}(\mathbb{B})=\Big\{u\in W_{{\rm loc}}^{m, p}({\rm int}~\mathbb{B});\omega u\in\mathcal{H}_{p}^{m, \gamma}(X^{\wedge})\Big\} \end{equation*} $

for any cut off function $\omega$, supported by a collar neighbourhood of $[0, 1)\times\partial\mathbb{B}$. Moreover, the subspace $\mathcal{H}_{p, 0}^{m, \gamma}(\mathbb{B})$ of $\mathcal{H}_{p}^{m, \gamma}(\mathbb{B})$ is defined as follows

$ \begin{equation*} \mathcal{H}_{p, 0}^{m, \gamma}(\mathbb{B}):=[\omega]\mathcal{H}_{p, 0}^{m, \gamma}(X^{\wedge})+[1-\omega]W_{0}^{m, p}({\rm int}~\mathbb{B}), \end{equation*} $

where $W_{0}^{m, p}({\rm int}~\mathbb{B})$ denotes the closure of $C_{0}^{\infty}({\rm int}~\mathbb{B})$ in Sobolev spaces $W^{m, p}(\tilde{X})$ when $\tilde{X}$ is a closed compact $C^{\infty}$ manifold of dimension of $N$ that containing $\mathbb{B}$ as a submanifold with boundary. More details on the properties of the spaces $\mathcal{H}_{p, 0}^{m, \gamma}(\mathbb{B})$ and $\mathcal{H}_{p}^{m, \gamma}(\mathbb{B})$ can be found in [10].

Next, we will recall the cone Sobolev inequality and Poincaré inequality. For details we refer to [12].

Lemma 2.1  (Cone Sobolev inequality) Assume that $1<p<N, ~\frac{1}{p^{*}}=\frac{1}{p}-\frac{1}{N}$, and $\gamma\in \mathbb{R}$. The following estimate

$ \begin{equation*} \|u\|_{L_{p^{*}}^{\gamma^{*}}(\mathbb{B})}\leq c_{1}\|(x_{1}\partial_{x_{1}})u\|_{L_{p}^{\gamma}(\mathbb{B})}+(c_{1}+c_{2})\sum\limits_{i=2}^{N}\|\partial_{x_{i}}u\|_{L_{p}^{\gamma}(\mathbb{B})}+c_{2}\|u\|_{L_{p}^{\gamma}(\mathbb{B})} \end{equation*} $

holds for all $u\in C_{0}^{\infty}(\mathbb{B})$, where $\gamma^{*}=\gamma-1, c_{1}=\frac{(N-1)p}{N(N-p)}, c_{2}=\frac{|N-\frac{(\gamma-1)(N-1)p}{N-p}|^{\frac{1}{N}}}{N}$. Moreover, if $u\in\mathcal{H}_{p, 0}^{1, \gamma}(\mathbb{B})$, we have $\|u\|_{L_{p^{*}}^{\gamma^{*}}(\mathbb{B})}\leq c\|u\|_{\mathcal{H}_{p}^{1, \gamma}(\mathbb{B})}, $ where the constant $c=c_{1}+c_{2}$ and $c_{1}, c_{2}$ are given.

Lemma 2.2  (Poincaré inequality). Let $\mathbb{B}=[0, 1)\times X$ ba a bounded subset in $\mathbb{R}_{+}^{N}$, and $1<p<+\infty, \gamma\in \mathbb{R}$. If $u(x_{1}, x^{\prime})\in\mathcal{H}_{p, 0}^{1, \gamma}(\mathbb{B})$, then $\|u(x_{1}, x^{\prime})\|_{L_{p}^{\gamma}(\mathbb{B})}\leq c\|\nabla_{\mathbb{B}}u(x_{1}, x^{\prime})\|_{L_{p}^{\gamma}(\mathbb{B})}, $ where the positive constant $c$ depending only on $\mathbb{B}$ and $p$.

Lemma 2.3  For $2<p<2^{*}$, the embedding $\mathcal{H}_{2, 0}^{1, N/2}(\mathbb{B})\hookrightarrow\mathcal{H}_{p, 0}^{0, N/p}(\mathbb{B})$ is compact. Then we set

$ \begin{align*} l_{1}(d, g)&:=\mu_{1}d^{p-1}+\beta d^{\frac{p}{2}-1}g^{\frac{p}{2}}-1, d>0, g>0;\\ l_{2}(d, g)&:=\mu_{2}g^{p-1}+\beta g^{\frac{p}{2}-1}d^{\frac{p}{2}}-1, g>0, d>0; \end{align*} $

Lemma 2.4  Suppose that $\beta\geq(p-1)\max\{\mu_{1}, \mu_{2}\}$. Then the following system

$ \begin{cases} d+g\leq d_{0}+d_{0}, \\ l_{1}(d, g)\geq0, l_{2}(d, g)\geq0, \\ d>0, g\geq0, (d, g)\neq(0, 0) \end{cases} $

(2.1)

has a unique solution $(d_{0}, g_{0})$.

Proof  See [1, Lemmas 2.1, 2.2, 2.3, 2.4].

Now we consider the solution of (1.2), we will prove that this solution is also a least energy solution.

Lemma 2.5  Assume that $-\lambda_{1}(\mathbb{B})<\lambda<0$, and then (1.2) has a positive least energy solution $\varphi\in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\mathbb{B})$ with energy

$ \begin{equation*} A_{1}:=\frac{1}{N}\int_{\mathbb{B}}(|\nabla_{\mathbb{B}}\varphi|^{2}+\lambda\varphi^{2})\frac{dx_{1}}{x_{1}}dx^{'}. \end{equation*} $

Proof  Let $S_{\lambda}(u;\mathbb{B})= \frac{\parallel \nabla_{\mathbb{B}}u \parallel_{L_{2}^{\frac{N}{2}}}^{2}+\lambda\parallel u\parallel_{L_{2}^{\frac{N}{2}}}^{2}}{\parallel u\parallel_{L_{2^{*}}^{\frac{N}{2^{*}}}}^{2}}$ and $ S_{\lambda}(\mathbb{B})=\inf\limits_{u\in H_{2, 0}^{1, \frac{N}{2}}(\mathbb{B}), u\neq0}S_{\lambda}(u;\mathbb{B})$. Set $C_{0}=\frac{1}{N}[S_{\lambda}(\mathbb{B})]^{\frac{N}{2}}$ and the functional

$ \begin{equation*} f_{\lambda}(u)=\frac{1}{2}\int_{\mathbb{B}}| \nabla_{\mathbb{B}}u|^{2}\frac{dx_{1}}{x_{1}}dx^{'}+\frac{\lambda}{2}\int_{\mathbb{B}} u^{2}\frac{dx_{1}}{x_{1}}dx^{'}-\frac{1}{2^{*}}\int_{\mathbb{B}}| u|^{2^{*}}\frac{dx_{1}}{x_{1}}dx^{'}. \end{equation*} $

From the result in [2], we know that (1.2) has a positive solution with energy $C_{0}$. Furthermore, we will show that $C_{0}$ is the least energy of (1.2). We set

$ \begin{equation*} \overline{\mathcal{N}}=\{u\in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\mathbb{B});\int_{\mathbb{B}}\left(| \nabla_{\mathbb{B}}u|^{2}+\lambda u^{2}\right)\frac{dx_{1}}{x_{1}}dx^{'}=\int_{\mathbb{B}}| u|^{2^{*}}\frac{dx_{1}}{x_{1}}dx^{'}\}. \end{equation*} $

If $u$ is the solution of (1.2), then $u\in\overline{\mathcal{N}}$ and $f_{\lambda}(u)=\frac{1}{N}\| u\|_{L_{2^{*}}^{\frac{N}{2^{*}}}}^{2^{*}}=\frac{1}{N}[S_{\lambda}(u;\mathbb{B})]^{\frac{N}{2}}$. If we denote $\inf\limits_{u\in\overline{\mathcal{N}}, u\neq0}f_{\lambda}(u)$ as the least energy of (1.2), then

$ \begin{equation*} \inf\limits_{u\in\overline{\mathcal{\mathcal{N}}}, u\neq0}f_{\lambda}(u)=\frac{1}{N}[\inf\limits_{u\in\overline{\mathcal{N}}, u\neq0}S_{\lambda}(u;\mathbb{B})]^{\frac{N}{2}} \geq \frac{1}{N}[S_{\lambda}(\mathbb{B})]^{\frac{N}{2}}=C_{0}. \end{equation*} $

Therefore $C_{0}=A_{1}=\inf\limits_{u\in\overline{\mathcal{N}}, u\neq0}f_{\lambda}(u)$. Let $\varphi$ be a positive critical point of $f_{\lambda}(u)$ with a critical value $A_{1}$. Then it is easy to get $A_{1}:=\frac{1}{N}\displaystyle\int_{\mathbb{B}}(|\nabla_{\mathbb{B}}\varphi|^{2}+\lambda\varphi^{2}) \frac{dx_{1}}{x_{1}}dx^{'}$.

In this section, we will prove Theorem 1.1 and Theorem 1.2. In particular, we will separate the proof of Theorem 1.2 into several steps.

Proof of Theorem 1.1  For $-\lambda_{1}(\mathbb{B})<\lambda_{1}=\lambda_{2}=\lambda<0$, we can easily get that $A=\inf\limits_{(u, v)\in\mathcal{N}}J(u, v)>0$. $\beta>0$, so (1.3) has a solution $(d_{0}, g_{0})$. By Lemma 2.5, we obtain that $\displaystyle\int_{\mathbb{B}}(|\nabla_{\mathbb{B}}\varphi|^{2}+\lambda\varphi^{2})\frac{dx_{1}}{x_{1}}dx^{'}=\displaystyle\int_{\mathbb{B}} \varphi^{2^{*}}\frac{dx_{1}}{x_{1}}dx^{'}$. For a direct computing, we can get that $(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)$ is a positive solution of (1.1). Moreover, we have

$ \begin{equation*} 0<A\leq J(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)=(d_{0}+g_{0})A_{1}. \end{equation*} $

(3.1)

Now if $\beta\geq(p-1)\max\{\mu_{1}, \mu_{2}\}$, then we have $A= J(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)$. In fact, we can take a minimizing sequence$\{(u_{n}, v_{n})\}_{n\in \mathbb{N}}\subset\mathcal{N}$ for $A$ such that $J(u_{n}, v_{n})\rightarrow A$. Then we get

$ \begin{align*} (NA_{1})^{\frac{2}{N}}c_{n}&\leq\int_{\mathbb{B}}(| \nabla_{\mathbb{B}}u_{n}|^{2}+\lambda u_{n}^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &=\int_{\mathbb{B}}(\mu_{1}| u_{n}|^{2p}+\beta | u_{n}|^{p}| v_{n}|^{p})\frac{dx_{1}}{x_{1}}dx^{'} \leq\mu_{1}c_{n}^{p}+\beta c_{n}^{\frac{p}{2}}k_{n}^{\frac{p}{2}} \end{align*} $

(3.2)

and

$ \begin{align*} (NA_{1})^{\frac{2}{N}}k_{n}&\leq\int_{\mathbb{B}}(| \nabla_{\mathbb{B}}v_{n}|^{2}+\lambda v_{n}^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &=\int_{\mathbb{B}}(\mu_{2}| v_{n}|^{2p}+\beta | u_{n}|^{p}| v_{n}|^{p})\frac{dx_{1}}{x_{1}}dx^{'} \leq\mu_{2}k_{n}^{p}+\beta c_{n}^{\frac{p}{2}}k_{n}^{\frac{p}{2}}, \end{align*} $

(3.3)

where $c_{n}=(\displaystyle\int_{\mathbb{B}}| u_{n}|^{2p}\frac{dx_{1}}{x_{1}}dx^{'})^{\frac{1}{p}}$ and $k_{n}=(\displaystyle\int_{\mathbb{B}}| v_{n}|^{2p}\frac{dx_{1}}{x_{1}}dx^{'})^{\frac{1}{p}}$. Note that

$ \begin{equation*} J(u_{n}, v_{n})=\frac{1}{N}\int_{\mathbb{B}}(| \nabla_{\mathbb{B}}u_{n}|^{2}+\lambda u_{n}^{2}+| \nabla_{\mathbb{B}}v_{n}|^{2}+\lambda v_{n}^{2})\frac{dx_{1}}{x_{1}}dx^{'}, \end{equation*} $

and then from (3.1), we have

$ (NA_{1})^{\frac{2}{N}}(c_{n}+k_{n})\leq N J(u_{n}, v_{n})\leq N (d_{0}+g_{0})A_{1}+o(1), $

(3.4)

$ \mu_{1}c_{n}^{p-1}+\beta c_{n}^{\frac{p}{2}-1}k_{n}^{\frac{p}{2}}\geq(NA_{1})^{\frac{2}{N}}, $

(3.5)

$ \mu_{2}k_{n}^{p-1}+\beta c_{n}^{\frac{p}{2}}k_{n}^{\frac{p}{2}-1}\geq(NA_{1})^{\frac{2}{N}}. $

(3.6)

Therefore, the sequences $\{c_{n}\}_{n\in \mathbb{N} }, \{k_{n}\}_{n\in \mathbb{N}}$ are uniformly bounded. Passing to a subsequence, we assume that $c_{n}\rightarrow c$ and $k_{n}\rightarrow k$ as $n\rightarrow \infty$ for some $c\geq0, k\geq0$. By (3.2) and (3.3), we have $\mu_{1}c^{p}+2\beta c^{\frac{p}{2}}k^{\frac{p}{2}}+\mu_{2}k^{p}\geq NA>0$. That means $c$ and $k$ are not necessary to be all vanished. From (3.4)-(3.6), we get

$ \begin{cases} \frac{c}{(NA_{1})^{1-\frac{2}{N}}}+\frac{k}{(NA_{1})^{1-\frac{2}{N}}}\leq d_{0}+g_{0}, \\ \mu_{1}\big[\frac{c}{(NA_{1})^{1-\frac{2}{N}}}\big]^{p-1} +\beta\big[\frac{c}{(NA_{1})^{1-\frac{2}{N}}}\big]^{\frac{p}{2}-1}\big[\frac{k}{(NA_{1})^{1-\frac{2}{N}}}\big]^{\frac{p}{2}}\geq1, \\ \mu_{2}\big[\frac{k}{(NA_{1})^{1-\frac{2}{N}}}\big]^{p-1} +\beta\big[\frac{c}{(NA_{1})^{1-\frac{2}{N}}}\big]^{\frac{p}{2}}\big[\frac{k}{(NA_{1})^{1-\frac{2}{N}}}\big]^{\frac{p}{2}-1}\geq1. \end{cases} $

(3.7)

Applying Lemma 2.4, we have

$ \begin{equation*} d_{0}=\frac{c}{(NA_{1})^{1-\frac{2}{N}}}, ~~ g_{0}=\frac{k}{(NA_{1})^{1-\frac{2}{N}}}, \end{equation*} $

here we get $c_{n}\rightarrow d_{0}(NA_{1})^{1-\frac{2}{N}}$ and $k_{n}\rightarrow g_{0}(NA_{1})^{1-\frac{2}{N}}$ as $n\rightarrow \infty$, and moreover,

$ \begin{equation*} NA=\lim\limits_{n\rightarrow\infty}NJ(u_{n}, v_{n})\geq\lim\limits_{n\rightarrow\infty}(NA_{1})^{\frac{2}{N}}(c_{n}+k_{n}) =N(d_{0}+g_{0})A_{1}. \end{equation*} $

That is, $A\geq(d_{0}+g_{0})A_{1}=J(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)$, and so $A=J(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)=(d_{0}+g_{0})A_{1}$. This tells us that $(\sqrt{d_{0}}\varphi, \sqrt{g_{0}}\varphi)$ is a positive least energy solution of (1.1).

Next we start to prove Theorem 1.2.

Lemma 3.1  For $-\infty<\beta<0$, if $B$ is attained by a couple $(u, v)\in\mathcal{M}$, then this couple is a critical point of $E(u, v)$ in (1.7). The proof is analogous to that in [1, Lemma 2.5]. So we omit it here.

By Lemma 2.1, let $S$ be the sharp constant of $D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N})\hookrightarrow L_{2^{*}}^{\frac{N}{2^{*}}}(\mathbb{R}_{+}^{N}), $

$ \begin{equation*} \int_{\mathbb{R}_{+}^{N}}| \nabla_{\mathbb{B}}u|^{2}\frac{dx_{1}}{x_{1}}dx^{'}\geq S(\int_{\mathbb{R}_{+}^{N}}| u|^{2^{*}}\frac{dx_{1}}{x_{1}}dx^{'})^{\frac{2}{2^{*}}}. \end{equation*} $

(3.8)

For $ \varepsilon>0$, let

$ \begin{equation*} U_{\varepsilon}(x_{1}, x^{'})=\frac{[N(N-2)\varepsilon^{2}]^{\frac{N-2}{4}}}{(\varepsilon^{2}+| {\rm ln}x_{1}|^{2}+| x^{'}|^{2})^{\frac{N-2}{2}}}. \end{equation*} $

(3.9)

Then $U_{\varepsilon}$ satisfies $-\Delta_\mathbb{B}u=| u|^{2^{*}-2}u$ in $\mathbb{R}_{+}^{N}$ (see [2, 5]). Moreover,

$ \begin{equation*} \int_{\mathbb{R}_{+}^{N}}| \nabla_{\mathbb{B}}U_{\varepsilon}|^{2}\frac{dx_{1}}{x_{1}}dx^{'}= \int_{\mathbb{R}_{+}^{N}}| U_{\varepsilon} |^{2^{*}}\frac{dx_{1}}{x_{1}}dx^{'}=S^{\frac{N}{2}}. \end{equation*} $

(3.10)

Now we give the proof of first part of Theorem 1.2.

Proof of (1) in Theorem 1.2 Let $\varphi_{\mu_{i}}:=\mu_{i}^{\frac{2-N}{4}}U_{1}$ with $U_{1}$ being as in (3.9). Then $\varphi_{\mu_{i}}$ satisfies the equation $-\Delta_\mathbb{B}u=\mu_{i}| u|^{2^{*}-2}u$ in $\mathbb{R}_{+}^{N}$. We set $e_{2}=(0, 1, 0, \cdots, 0)\in\mathbb{R}_{+}^{N}$ and $(u_{r}(x), v_{r}(x))=(\varphi_{\mu_{1}(x)}, \varphi_{\mu_{2}(x+re_{2})}).$ Then $v_{r}\rightharpoonup0$ weakly in $D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N})$ and $v^{p}_{r}\rightharpoonup0$ weakly in $L_{2}^{\frac{N}{2}}(\mathbb{R}_{+}^{N})$ as $r\rightarrow\infty$. That is,

$ \begin{equation*} \lim\limits_{r\rightarrow\infty}\int_{\mathbb{R}_{+}^{N}}u^{p}_{r}v^{p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}=0. \end{equation*} $

To complete this proof, we claim that: for $r>0$ sufficiently large and $\beta<0$, there exists $(t_{r}u_{r}, s_{r}v_{r})\in\mathcal{M}$ with $t_{r}>1, s_{r}>1$.

In fact, note that $u_{r}$ and $v_{r}$ satisfy the equation $-\Delta_\mathbb{B}u=\mu_{i}| u|^{2^{*}-2}u$. If $(tu_{r}, sv_{r})\in\mathcal{M}$, then we have

$ \begin{align*} t^{2}\int_{\mathbb{B}_{+}^{N}}\mu_{1} u_{r}^{2p}\frac{dx_{1}}{x_{1}}dx^{'} &=t^{2}\int_{\mathbb{B}_{+}^{N}}|\nabla_{\mathbb{B}} u_{r}|^{2}\frac{dx_{1}}{x_{1}}dx^{'}\\ &=t^{2p}\int_{\mathbb{B}_{+}^{N}}\mu_{1} u_{r}^{2p}\frac{dx_{1}}{x_{1}}dx^{'} +t^{p}s^{p}\int_{\mathbb{B}_{+}^{N}}\beta |u_{r}|^{p}| v_{r}|^{p}\frac{dx_{1}}{x_{1}}dx^{'} \end{align*} $

and

$ \begin{align*} s^{2}\int_{\mathbb{B}_{+}^{N}}\mu_{2} v_{r}^{2p}\frac{dx_{1}}{x_{1}}dx^{'} &=s^{2}\int_{\mathbb{B}_{+}^{N}}|\nabla_{\mathbb{B}} v_{r}|^{2}\frac{dx_{1}}{x_{1}}dx^{'}\\ &=s^{2p}\int_{\mathbb{B}_{+}^{N}}\mu_{2} v_{r}^{2p}\frac{dx_{1}}{x_{1}}dx^{'} +t^{p}s^{p}\int_{\mathbb{B}_{+}^{N}}\beta | u_{r}|^{p}| v_{r}|^{p}\frac{dx_{1}}{x_{1}}dx^{'}. \end{align*} $

Since $v_{r}(x)\rightarrow0~~ (r\rightarrow\infty)$, there exists $\delta_{r}>0$ for $r$ sufficiently large such that $v_{r}(x)\leq\delta_{r}$ and $\lim\limits_{r\rightarrow\infty}\delta_{r}=0$. By cone Sobolev inequality, we obtain that for some $C>0$,

$ \begin{align*} \big(\int_{\mathbb{R}_{+}^{N}}\beta u^{p}_{r}v^{p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}\big)^{2} &\leq\beta^{2}\delta_{r}^{2p-2} \big(\int_{\mathbb{R}_{+}^{N}}u^{p}_{r}v\frac{dx_{1}}{x_{1}}dx^{'}\big)^{2}\\ &\leq\beta^{2}\delta_{r}^{2p-2}\int_{\mathbb{R}_{+}^{N}} u^{2p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}\int_{\mathbb{R}_{+}^{N}} v_{r}^{2}\frac{dx_{1}}{x_{1}}dx^{'}\\ &\leq C\beta^{2}\delta_{r}^{2p-2}\int_{\mathbb{R}_{+}^{N}}\mu_{1} u^{2p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}\int_{\mathbb{R}_{+}^{N}} \mu_{2}v_{r}^{2p}\frac{dx_{1}}{x_{1}}dx^{'}\\ &<\int_{\mathbb{R}_{+}^{N}}\mu_{1} u^{2p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}\int_{\mathbb{R}_{+}^{N}} \mu_{2}v_{r}^{2p}\frac{dx_{1}}{x_{1}}dx^{'}. \end{align*} $

For simplicity, we denote

$ \begin{align*} &D_{1}:=\mu_{1}\int_{\mathbb{R}_{+}^{N}} u^{2p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}=\mu_{1}\int_{\mathbb{R}_{+}^{N}} \varphi_{\mu_{1}}^{2p}\frac{dx_{1}}{x_{1}}dx^{'}>0, \\ &D_{2}:=\mu_{2}\int_{\mathbb{R}_{+}^{N}} v^{2p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}=\mu_{2}\int_{\mathbb{R}_{+}^{N}} \varphi_{\mu_{2}}^{2p}\frac{dx_{1}}{x_{1}}dx^{'}>0, \\ &F_{r}:=| \beta|\int_{\mathbb{R}_{+}^{N}}u^{p}_{r} v^{p}_{r}\frac{dx_{1}}{x_{1}}dx^{'}\rightarrow0, ~ {\rm as}~~r\rightarrow\infty. \end{align*} $

So $D_{1}D_{2}-F_{r}^{2}>0$. Recall that $(tu_{r}, sv_{r})\in\mathcal{M}$, and thus we get

$ \begin{cases} t^{2-p}D_{1}=t^{p}D_{1}+s^{p}F_{r}, \\ s^{2-p}D_{2}=s^{p}D_{2}+t^{p}F_{r}, ~~~t, s>0. \end{cases} $

(3.11)

From the first equality of (3.11), we obtain $s^{p}=(t^{2-p}-t^{p})\frac{D_{1}}{F_{r}}>0$, and therefore $t>1$. Similarly, we have $s>1$. Note that (3.11) is equivalent to $w(t)=0$, where

$ \begin{equation*} w(t)=D_{2}[\frac{D_{1}}{F_{r}}(t^{2-2p}-1)]^{\frac{2-p}{p}}+\frac{D_{1}D_{2}-F_{r}^{2}}{F_{r}}t^{2p-2} -\frac{D_{1}D_{1}}{F_{r}}. \end{equation*} $

For $1<p<2$, we get $w(1)=-F_{r}>0$, and $\lim\limits_{t\rightarrow\infty}w(t)<0$. So there exists $t_{r}>1$ such that $w(t)=0$.

Note that $(t_{r}u_{r}, s_{r}v_{r})\in\mathcal{M}$, and then we have

$ \begin{equation*} t_{r}^{2}D_{1}=t_{r}^{2p}D_{1}-t_{r}^{p}s_{r}^{p}F_{r}, s_{r}^{2}D_{2}=s_{r}^{2p}D_{2}-t_{r}^{p}s_{r}^{p}F_{r}. \end{equation*} $

(3.12)

Up to a subsequence, if $t_{r}\rightarrow\infty$ as $r\rightarrow\infty$, then by the fact

$ \begin{equation*} t_{r}^{2p}D_{1}- t_{r}^{2}D_{1}=s_{r}^{2p}D_{2}- s_{r}^{2}D_{2}, \end{equation*} $

we also get $t\rightarrow\infty ~~(r\rightarrow\infty)$. As $2-p<p$, for $r$ large enough, we have

$ \begin{equation*} t_{r}^{p}D_{1}- t_{r}^{2-p}D_{1}\geq\frac{1}{2}t_{r}^{p}D_{1}, s_{r}^{p}D_{2}- s_{r}^{2-p}D_{2}\geq\frac{1}{2}s_{r}^{p}D_{2}. \end{equation*} $

Therefore, we obtain

$ \begin{equation*} F_{r}=\frac{t^{p}_{r}-t^{2-p}_{r}}{s^{p}_{r}}D_{1}\geq\frac{t^{p}_{r}}{2s^{p}_{r}}D_{1}, F_{r}=\frac{s^{p}_{r}-s^{2-p}_{r}}{s^{p}_{r}}D_{2}\geq\frac{s^{p}_{r}}{2t^{p}_{r}}D_{2}. \end{equation*} $

This means that $0<\frac{1}{4}D_{1}D_{2}\leq F^{2}_{r}\rightarrow0, ~~{\rm as}~~r\rightarrow\infty, $ which is a contradiction. Hence $t_{r}$ and $s_{r}$ are uniformly bounded. By (3.12) and $F_{r}\rightarrow0 ~~(r\rightarrow\infty$), we have $\lim\limits_{r\rightarrow\infty}t_{r}=\lim\limits_{r\rightarrow\infty}s_{r}=1.$

For $(t_{r}u_{r}, s_{r}v_{r})\in\mathcal{M}$, from (3.10) we have

$ \begin{align*} B&\leq E(t_{r}u_{r}, s_{r}v_{r})\\ &=\frac{1}{N}(t^{2}_{r}\int_{\mathbb{R}_{+}^{N}}| \nabla_{\mathbb{B}}u_{r}|^{2}\frac{dx_{1}}{x_{1}}dx^{'} +s^{2}_{r}\int_{\mathbb{R}_{+}^{N}}| \nabla_{\mathbb{B}}v_{r}|^{2}\frac{dx_{1}}{x_{1}}dx^{'})\\ &=\frac{1}{N}(t^{2}_{r}\mu_{1}^{-\frac{N-2}{2}}+s^{2}_{r}\mu^{-\frac{N-2}{2}}_{2})S^{\frac{N}{2}}. \end{align*} $

Let $r\rightarrow\infty$, we get that $B\leq\frac{1}{N}(\mu_{1}^{-\frac{N-2}{2}}+\mu^{-\frac{N-2}{2}}_{2})S^{\frac{N}{2}}$.

On the other hand, for any $(u, v)\in\mathcal{M}$, by the fact $\beta<0$ and (3.8), we get that

$ \begin{equation*} \int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u|^{2}\frac{dx_{1}}{x_{1}}dx^{'} \leq\mu_{1}\int_{\mathbb{R}_{+}^{N}}| u|^{2p}\frac{dx_{1}}{x_{1}}dx^{'} \leq\mu_{1}S^{-p}(\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u|^{2}\frac{dx_{1}}{x_{1}}dx^{'})^{p}. \end{equation*} $

Therefore $\displaystyle\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u|^{2}\frac{dx_{1}}{x_{1}}dx^{'}\geq \mu_{1}^{-\frac{N-2}{2}}S^{\frac{N}{2}}$, and similarly, $\displaystyle\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} v|^{2}\frac{dx_{1}}{x_{1}}dx^{'}\geq \mu_{2}^{-\frac{N-2}{2}}S^{\frac{N}{2}}$. Note that $B=\inf\limits_{(u, \nu)\in\mathcal{M}}\{\frac{1}{N}\displaystyle\int_{\mathbb{R}_{+}^{N}}(| \nabla_{\mathbb{B}}u|^{2}+| \nabla_{\mathbb{B}}v|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\}$, and then we obtain that $B\geq\frac{1}{N}(\mu_{1}^{-\frac{N-2}{2}}+\mu^{-\frac{N-2}{2}}_{2})S^{\frac{N}{2}}$. Hence $B=\frac{1}{N}(\mu_{1}^{-\frac{N-2}{2}}+\mu^{-\frac{N-2}{2}}_{2})S^{\frac{N}{2}}.$

Now if $B$ is attained by some $(u, v)\in\mathcal{M}$, then $(|u|, |v|)\in\mathcal{M}$ and $E(|u|, |v|)=B$. From Lemma 3.1, we know that $(| u|, |v|)$ is a nontrivial solution of (1.6). By the maximum principle, we may assume that $u>0, v>0$, and so $\displaystyle\int_{\mathbb{R}_{+}^{N}} u^{p}v^{p}\frac{dx_{1}}{x_{1}}dx^{'}>0$. Moreover, we get

$ \begin{equation*} \int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u|^{2}\frac{dx_{1}}{x_{1}}dx^{'}<\mu_{1}\int_{\mathbb{R}_{+}^{N}}| u|^{2p}\frac{dx_{1}}{x_{1}}dx^{'} \leq\mu_{1}S^{-p}(\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u|^{2}\frac{dx_{1}}{x_{1}}dx^{'})^{p}. \end{equation*} $

It is easy to see that

$ \begin{equation*} B=E(u, v)=\frac{1}{N}\int_{\mathbb{R}_{+}^{N}}(|\nabla_{\mathbb{B}} u|^{2}+|\nabla_{\mathbb{B}} v|^{2})\frac{dx^{1}}{x^{1}}dx^{'}>\frac{1}{N}(\mu_{1}^{-\frac{N-2}{2}}+\mu^{-\frac{N-2}{2}}_{2})S^{\frac{N}{2}}, \end{equation*} $

that is a contradiction. We complete the proof.

Now we begin to prove (2-1) in Theorem 1.2.

Proof of (2-1) in Theorem 1.2  For $\beta>0$, $(\sqrt{d_{0}}U_{\varepsilon}, \sqrt{g_{0}}U_{\varepsilon})$ is a nontrivial solution of (1.6) and $B\leq E(\sqrt{d_{0}}U_{\varepsilon}, \sqrt{g_{0}}U_{\varepsilon})=\frac{1}{N}(d_{0}+g_{0})S^{\frac{N}{2}}$.

We let $\beta\geq(p-1)\max\{\mu_{1}, \mu_{2}\}$ and $\{(u_{n}, v_{n})\}_{n\in \mathbb{N}}\subset\mathcal{M}$ be a minimizing sequence for $B$, that is, $E(u_{n}, v_{n})\rightarrow B$. Define $c_{n}=(\displaystyle\int_{\mathbb{R}_{+}^{N}}| u_{n}|^{2p}\frac{dx_{1}}{x_{1}}dx^{'})^{\frac{1}{p}}, k_{n}=(\displaystyle\int_{\mathbb{R}_{+}^{N}}| v_{n}|^{2p}\frac{dx_{1}}{x_{1}}dx^{'})^{\frac{1}{p}}$, and we have

$ \begin{eqnarray*} &&Sc_{n}\leq\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u_{n}|^{2p}\frac{dx_{1}}{x_{1}}dx^{'} =\int_{\mathbb{R}_{+}^{N}}(\mu_{1}| u_{n}|^{2p}+\beta| u_{n}|^{p}| v_{n}|^{p})\frac{dx_{1}}{x_{1}}dx^{'} \leq\mu_{1}c_{n}^{p}+\beta c_{n}^{\frac{p}{2}}d_{n}^{\frac{p}{2}}, \\ &&Sd_{n}\leq\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} v_{n}|^{2p}\frac{dx_{1}}{x_{1}}dx^{'} =\int_{\mathbb{R}_{+}^{N}}(\mu_{2}| v_{n}|^{2p}+\beta| u_{n}|^{p}| v_{n}|^{p})\frac{dx_{1}}{x_{1}}dx^{'} \leq\mu_{2}d_{n}^{p}+\beta c_{n}^{\frac{p}{2}}d_{n}^{\frac{p}{2}}, \end{eqnarray*} $

which imply

$ \begin{eqnarray*} &&S(c_{n}+d_{n})\leq NE(u_{n}, v_{n})\leq(d_{0}+g_{0})S^{\frac{N}{2}}+o(1), \\ &&\mu_{1}c_{n}^{p-1}+\beta c_{n}^{\frac{p}{2}-1}k_{n}^{\frac{p}{2}}\geq S, ~~ \mu_{2}k_{n}^{p-1}+\beta k_{n}^{\frac{p}{2}-1}c_{n}^{\frac{p}{2}}\geq S. \end{eqnarray*} $

Similarly as in the proof of Theorem 1.1, we have $c_{n}\rightarrow d_{0}S^{\frac{N}{2}-1}, d_{n}\rightarrow g_{0}S^{\frac{N}{2}-1} ~~(n\rightarrow\infty).$ Moreover, we obtain

$ \begin{equation*} NB=\lim\limits_{n\rightarrow\infty}NE(u_{n}, \nu_{n})\geq \lim\limits_{n\rightarrow\infty}S(c_{n}+k_{n}) =(d_{0}+g_{0})S^{\frac{N}{2}}. \end{equation*} $

Since $B\leq \frac{1}{N}(d_{0}+g_{0})S^{\frac{N}{2}}$, we obtain that

$ \begin{equation*} B=\frac{1}{N}(d_{0}+g_{0})S^{\frac{N}{2}}=E(\sqrt{d_{0}}U_{\varepsilon}, \sqrt{g_{0}}U_{\varepsilon}). \end{equation*} $

Therefore $(\sqrt{d_{0}}U_{\varepsilon}, \sqrt{g_{0}}U_{\varepsilon})$ is a positive least energy solution of (1.6).

Next we continue the proof of (2-2) in Theorem 1.2. For this purpose we need to show that (1.6) has a positive least energy solution for any $0<\beta<(p-1)\max\{\mu_{1}, \mu_{2}\}$. Therefore, we assume $\beta>0$, and define $B^{'}:=\inf\limits_{(u, v)\in\mathcal{M}^{'}}E(u, v), $ where

$ \begin{equation*} \mathcal{M}^{'}:=\{(u, v)\in D\backslash\{(0, 0)\}, E'(u, v)(u, v)=0\}. \end{equation*} $

It is easy to see that $\mathcal{M}\subset \mathcal{M}^{'}$, and so $B^{'}\leq B$. By cone Sobolev inequality, we have $B^{'}>0$. We set $\Omega_{R}(1, 0):=\{(x_{1}, x^{'})\in \mathbb{R}_{+}^{N};(\ln x_{1})^{2}+| x^{'}|^{2}<R^{2}\}, H(x_{0}, R):=\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{R}(x_{0}))\times \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{R}(x_{0}))$ for $x_{0}=(1, 0, \cdots, 0)\in\mathbb{R}_{+}^{N}$. Consider the system

$ \begin{cases} -\Delta_{\mathbb{B}}u=\mu_{1}| u|^{2p-2}u+\beta| u|^{p-2}u|v|^{p}, x \in \Omega_{R}(x_{0}), \\ -\Delta_{\mathbb{B}}v=\mu_{2}| v| ^{2p-2}v+\beta | v| ^{p-2}v | u| ^{p}, x \in \Omega_{R}(x_{0}) , \\ u, v \in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{R}(x_{0})), \end{cases} $

(3.13)

and define $B^{'}(R):=\inf\limits_{(u, v)\in\mathcal{M}^{'}(R)}E(u, v), $ where

$ \begin{eqnarray*} \mathcal{M}^{'}(R)&:=&\{(u, v)\in H(x_{0}, R)\setminus\{(0, 0)\}, \displaystyle\int_{\Omega_{R}(x_{0})}(|\nabla_{\mathbb{B}} u|^{2}+|\nabla_{\mathbb{B}} v|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &&-\displaystyle\int_{\Omega_{R}(x_{0})}(\mu_{1}| u|^{2p}+2\beta| u|^{p}| v|^{p}+\mu_{2}| v|^{2p})\frac{dx_{1}}{x_{1}}dx^{'}=0\}. \end{eqnarray*} $

Lemma 3.2  For all $R>0$, we have $B^{'}(R)\equiv B^{'}$.

Proof Let $R_{1}>R_{2}$, since $\mathcal{M}^{'}(R_{2})\subset\mathcal{M}^{'}(R_{1})$, we get $B(R_{1})\leq B^{'}(R_{2})$. For any $(u, v)\in\mathcal{M}^{'}(R_{1})$, we define

$ \begin{equation*} (u_{1}(x), v_{1}(x))=\left((\frac{R_{1}}{R_{2}})^{\frac{N-2}{2}}u(x^{\frac{R_{1}}{R_{2}}}_{1}, \frac{R_{1}}{R_{2}}x^{'}), (\frac{R_{1}}{R_{2}})^{\frac{N-2}{2}}v(x^{\frac{R_{1}}{R_{2}}}_{1}, \frac{R_{1}}{R_{2}}x^{'})\right). \end{equation*} $

It is easy to see that $(u_{1}, v_{1})\in\mathcal{M}^{'}(R_{2})$, and so

$ \begin{equation*} B^{'}(R_{2})\leq E(u_{1}, v_{1})=E(u, v) ~~{\rm for} (u, v)\in \mathcal{M}^{'}(R_{1}). \end{equation*} $

That is, $B^{'}(R_{2})\leq B^{'}(R_{1})$. Hence we have $B^{'}(R_{1})= B^{'}(R_{2})$.

Let $\{(u_{n}, v_{n})\}_{n\in\mathbb{N}}\subset\mathcal{M}^{'} $ be a minimizing sequence of $B^{'}$. Moreover, we may assume that $u_{n}, v_{n}\in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{R_{n}}(x_{0}))$ for some $R_{n}>0$. Then $(u_{n}, v_{n})\in\mathcal{M}^{'}(R_{n}) $ and

$ \begin{equation*} B^{'}=\lim\limits_{n\rightarrow\infty}E(u_{n}, v_{n})\geq\lim\limits_{n\rightarrow\infty}B^{'}(R_{n})\equiv B^{'}(R). \end{equation*} $

Note that $B^{'}\leq B^{'}(R)$ and consequently we have $B^{'}(R)\equiv B^{'}$ for any $ R>0$.

Let $0\leq\varepsilon<p-1$. Consider

$ \begin{cases} -\Delta_{\mathbb{B}}u=\mu_{1}| u|^{2p-2-\varepsilon}u+\beta| u|^{p-2-\varepsilon}u|v|^{p-\varepsilon}, x \in \Omega_{1}(x_{0}), \\ -\Delta_{\mathbb{B}}v=\mu_{2}| v| ^{2p-2-\varepsilon}v+\beta |v|^{p-2-\varepsilon}v | u| ^{p-\varepsilon}, x \in \Omega_{1}(x_{0}) , \\ u, v \in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0})), \end{cases} $

(3.14)

and define $B_{\varepsilon}=\inf\limits_{(u, v)\in\mathcal{M}^{'}}E_{\varepsilon}(u, v)$, where

$ \begin{align*} E_{\varepsilon}(u, v)=&\frac{1}{2}\int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u|^{2}+| \nabla_{\mathbb{B}}v|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &-\frac{1}{2p-2\varepsilon}\int_{\Omega_{1}(x_{0})}(\mu_{1}| u|^{2p-2\varepsilon}+2\beta| u|^{p-\varepsilon}|v|^{p-\varepsilon}+\mu_{2}| v| ^{2p-2\varepsilon})\frac{dx_{1}}{x_{1}}dx^{'}. \end{align*} $

Set $\mathcal{M}^{'}_{\varepsilon}:=\{(u, v)\in H(x_{0}, R)\setminus{(0, 0)}, H_{\varepsilon}(u, v):=E^{'}_{\varepsilon}(u, v)(u, v)=0\}.$

Lemma 3.3  For $0<\varepsilon<p-1$, there holds

$ \begin{equation*} B_{\varepsilon}<\min\{ \inf\limits_{(u, 0)\in \mathcal{M}^{'}_{\varepsilon}}E_{\varepsilon}(u, 0), \inf\limits_{(0, v)\in \mathcal{M}^{'}_{\varepsilon}}E_{\varepsilon}(0, v)\}. \end{equation*} $

The proof is analogous to that in [1, Lemma 2.7]. So we omit it here.

Similarly as in Lemma 3.3, we have

$ \begin{align*} B^{'}&<\min \big\{ \inf\limits_{(u, 0)\in \mathcal{M}^{'}}E(u, 0), \inf\limits_{(0, v)\in \mathcal{M}^{'}}E(0, v)\big\}\\ &=\min\big\{ E(\varphi_{\mu_{1}}, 0), E(0, \varphi_{\mu_{2}})\big\} =\min\big\{\frac{1}{N}\mu_{1}^{-\frac{N-2}{2}}S^{\frac{N}{2}} , \frac{1}{N}\mu^{-\frac{N-2}{2}}_{2}S^{\frac{N}{2}} \big \}, \end{align*} $

where $\varphi_{\mu_{i}}$ is the same as in the proof of (1) in Theorem 1.2.

Now we introduce the "Cone Schwartz symmetrization". Assume that $\Omega$ is a bounded domain of $\mathbb{R}_{+}^{N}$ and $u$ is a real measurable function defined on $\Omega$. We define the distribution function of $u$ as follows $u_{\sharp}(t)={\rm meas} \{ x\in\Omega:|u(x)|>t\}~~{\rm for}~~t\in\mathbb{R}, $ where "meas" denotes the corresponding measure in cone Sobolev space. Then we can define the decreasing rearrangement of $u$ in the form $\widetilde{u}(s)=\inf\{t\in\mathbb{R}:u_{\sharp}(t)\leq s\}~{\rm for}~s\in[0, | \Omega|].$ We call $u^{*}(x)$ the cone Schwarz symmetrization of $u$ if $u^{*}(x)=\widetilde{u}(o_{n}| x|_{\mathbb{B}}^{N})~{\rm for}~x\in\widetilde{\Omega}, $ where $\widetilde{\Omega}$ is the sphere centred at $x_{0}$ with the same measure of $\Omega$, and $| x-z|_{\mathbb{B}}=(| \ln\frac{x_{1}}{z_{1}}|^{2}+| x^{'}-z^{'}|^{2})^{\frac{1}{2}}$ for $x=(x_{1}, x^{'})$, $z=(z_{1}, z^{'})$, here $o_{n}$ is the measure of the unit ball in $\mathbb{R}_{+}^{N}$. Since $\widetilde{u}$ is decreasing, $u^{*}$ is partly radially symmetric decreasing in relation to $ | x|_{\mathbb{B}}$.

Lemma 3.4  For any $0<\varepsilon<p-1$, (3.14) has a classical least energy solution $(u_{\varepsilon}, v_{\varepsilon})$, and $u_{\varepsilon}, v_{\varepsilon}$ are both partly radially symmetric decreasing.

Proof  Fix any $0<\varepsilon<p-1$, and then it is easy to see that $B_{\varepsilon}>0$. Let $(u, v)\in\mathcal{M}^{'}_{\varepsilon}$ with $u\geq0, v\geq0$, and $(u^{*}, v^{*})$ be its cone Schwartz symmetrization. Then we have

$ \begin{align*} &\int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u^{*}|^{2}+| \nabla_{\mathbb{B}}v^{*}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ \leq&\int_{\Omega_{1}(x_{0})}(\mu_{1}| u^{*}|^{2p-2\varepsilon}+2\beta| u^{*}|^{p-\varepsilon}|v^{*}|^{p-\varepsilon}+\mu_{2}|v| ^{2p-2\varepsilon})\frac{dx_{1}}{x_{1}}dx^{'}. \end{align*} $

Similarly as in Lemma 3.3, there exists $0 < t^{*}\leq1$ such that $(t^{*}u^{*}, t^{*}v^{*})\in\mathcal{M}^{'}_{\varepsilon}$, and then we get

$ \begin{align*} E_{\varepsilon}(t^{*}u^{*}, t^{*}v^{*})&=(\frac{1}{2}-\frac{1}{2p-2\varepsilon})(t^{*})^{2}\int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u^{*}|^{2}+| \nabla_{\mathbb{B}}v^{*}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &\leq(\frac{1}{2}-\frac{1}{2p-2\varepsilon})\int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u|^{2}+| \nabla_{\mathbb{B}}v|^{2})\frac{dx_{1}}{x_{1}}dx^{'}=E_{\varepsilon}(u, v). \end{align*} $

(3.15)

We take a minimizing sequence $\{(u_{n}, v_{n})\}_{n\in \mathbb{N}}\subset \mathcal{M}^{'}_{\varepsilon}$ with $u_{n}\geq0, v_{n}\geq0$ such that $E_{\varepsilon}(u_{n}, v_{n})\rightarrow B_{\varepsilon}$. Let $(u_{n}^{*}, v_{n}^{*})$ be its "cone Schwartz symmetrization". Then there exists $0 < t^{*}_{n}\leq1$ such that $(t^{*}_{n}u^{*}_{n}, t^{*}_{n}v^{*}_{n})\in\mathcal{M}^{'}_{\varepsilon}$. By (3.15), we get

$ \begin{align*} B_{\varepsilon}\leq E_{\varepsilon}(t^{*}_{n}u^{*}_{n}, t^{*}_{n}v^{*}_{n})\leq(t^{*}_{n})^{2}E_{\varepsilon}(u_{n}, v_{n}) \leq E_{\varepsilon}(u_{n}, v_{n}). \end{align*} $

Therefore, we obtain $t^{*}_{n}\rightarrow 1, E_{\varepsilon}(u_{n}^{*}, v_{n}^{*})\rightarrow B_{\varepsilon}$, as $n\rightarrow\infty$, and $u_{n}^{*}, v_{n}^{*}$ are bounded in $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))$. Passing to a subsequence, we may assume that $u_{n}^{*}\rightharpoonup u_{\varepsilon}, v_{n}^{*}\rightharpoonup v_{\varepsilon}$ weakly in $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))$. By the compactness of the embedding $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))\hookrightarrow L^{\frac{N}{2p-2\varepsilon}}_{2p-2\varepsilon}(\Omega_{1}(x_{0})$ and $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))\hookrightarrow L^{\frac{N}{p-\varepsilon}}_{p-\varepsilon}(\Omega_{1}(x_{0})$, we have

$ \begin{align*} &\int_{\Omega_{1}(x_{0})}(\mu_{1}| u_{\varepsilon}|^{2p-2\varepsilon}+2\beta| u_{\varepsilon}|^{p-\varepsilon}|v_{\varepsilon}|^{p-\varepsilon}+\mu_{2}| v_{\varepsilon}| ^{2p-2\varepsilon})\frac{dx_{1}}{x_{1}}dx^{'}\\ =&\frac{2p-2\varepsilon}{p-1-\varepsilon}\lim\limits_{n\rightarrow\infty}E_{\varepsilon}(u_{n}^{*}, v_{n}^{*}) =\frac{2p-2\varepsilon}{p-1-\varepsilon}B_{\varepsilon}>0, \end{align*} $

which means $(u_{\varepsilon}, v_{\varepsilon})\neq(0, 0)$. Moreover, $u_{\varepsilon}\geq0, v_{\varepsilon}\geq0$ are partly radially symmetric.

Meanwhile, since

$ \int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u_{\varepsilon}|^{2}+| \nabla_{\mathbb{B}}v_{\varepsilon}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\leq \lim\limits_{n\rightarrow\infty}\int_{\Omega_{1}(x_{0})}(|\nabla_{\mathbb{B}}u_{n}^{*}|^{2}+| \nabla_{\mathbb{B}}v_{n}^{*}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}, $

we get

$ \begin{align*} &\int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u_{\varepsilon}|^{2}+| \nabla_{\mathbb{B}}v_{\varepsilon}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ \leq& \int_{\Omega_{1}(x_{0})}(\mu_{1}| u_{\varepsilon}|^{2p-2\varepsilon}+2\beta| u_{\varepsilon}|^{p-\varepsilon}|v_{\varepsilon}|^{p-\varepsilon}+\mu_{2}| v_{\varepsilon}| ^{2p-2\varepsilon})\frac{dx_{1}}{x_{1}}dx^{'}. \end{align*} $

Therefore, there exists $0 < t_{\varepsilon}\leq1$ such that $(t_{\varepsilon}u_{\varepsilon}, t_{\varepsilon}v_{\varepsilon})\in \mathcal{M}^{'}_{\varepsilon}$, and then

$ \begin{align*} B_{\varepsilon}\leq E_{\varepsilon}(t_{\varepsilon}u_{\varepsilon}, t_{\varepsilon}v_{\varepsilon}) &=(t_{\varepsilon})^{2}(\frac{1}{2}-\frac{1}{2p-2\varepsilon})\int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u_{\varepsilon}|^{2}+| \nabla_{\mathbb{B}}v_{\varepsilon}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &=(t_{\varepsilon})^{2}\lim\limits_{n\rightarrow\infty}E_{\varepsilon}(u_{n}^{*}, v_{n}^{*})=(t_{\varepsilon})^{2} B_{\varepsilon}\leq B_{\varepsilon}. \end{align*} $

That is $t_{\varepsilon}=1$ and $(u_{\varepsilon}, v_{\varepsilon})\in \mathcal{M}^{'}_{\varepsilon}$ with $E_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})=B_{\varepsilon}$. Therefore, $u_{n}^{*}\rightarrow u_{\varepsilon}, v_{n}^{*}\rightarrow v_{\varepsilon}$ strongly in $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))$ as $n\rightarrow\infty$.

By Lagrange multiplier theorem, we get that there exists a Lagrange multiplier $\tau\in\mathbb{R}$ such that $E^{'}_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})-\tau H^{'}_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})=0.$ Note that $E^{'}_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})(u_{\varepsilon}, v_{\varepsilon}) =H_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})=0$ and

$ \begin{align*} &H^{'}_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})(u_{\varepsilon}, v_{\varepsilon})\\ =&(2+2\varepsilon-2p)\int_{\Omega_{1}(x_{0})}(\mu_{1}| u_{\varepsilon}|^{2p-2\varepsilon}+2\beta| u_{\varepsilon}|^{p-\varepsilon}|v_{\varepsilon}|^{p-\varepsilon} +\mu_{2}|v_{\varepsilon}| ^{2p-2\varepsilon})\frac{dx_{1}}{x_{1}}dx^{'}. \end{align*} $

We get that $\tau=0$ and $E^{'}_{\varepsilon}(u_{\varepsilon}, \nu_{\varepsilon})=0$. By Lemma 3.3, we see that $u_{\varepsilon}\not\equiv0, v_{\varepsilon}\not\equiv0$. This means that $(u_{\varepsilon}, v_{\varepsilon})$ is a least energy solution of (3.14). By regularity theory and the maximum principle, we see that $u_{\varepsilon}>0, v_{\varepsilon}>0$ in $\Omega_{1}(x_{0}), u_{\varepsilon}, v_{\varepsilon}\in C^{2}(\Omega_{1}(x_{0}))$. This completes the proof.

Completion of the Proof of (2-2) in Theorem 1.2  For any $(u, v)\in\mathcal{M}^{'}(1)$, it is easy to see that there exists $t_{\varepsilon}>0$ such that $(t_{\varepsilon}u, t_{\varepsilon}v)\in\mathcal{M}^{'}_{\varepsilon}$ with $t_{\varepsilon}\rightarrow1 ~~(\varepsilon\rightarrow0)$, then

$ \begin{equation*} \limsup\limits_{\varepsilon\rightarrow0}B_{\varepsilon}\leq\limsup\limits_{\varepsilon\rightarrow0} E_{\varepsilon}(t_{\varepsilon}u, t_{\varepsilon}v)=E(u, v)~~{\rm for}~~ (u, v)\in\mathcal{M}^{'}(1). \end{equation*} $

By Lemma 3.2, we have

$ \begin{equation*} \limsup\limits_{\varepsilon\rightarrow0}B_{\varepsilon}\leq B^{'}(1)=B^{'}. \end{equation*} $

(3.16)

By Lemma 3.4, we know that there exists a positive least energy solution $(u_{\varepsilon}, v_{\varepsilon})$ of (3.14), which is partly radically symmetric decreasing. Recall that $E^{'}_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})(u_{\varepsilon}, v_{\varepsilon})=0$. By cone Sobolev inequality, we have

$ \begin{equation*} \frac{2p-2\varepsilon}{p-\varepsilon-1}B_{\varepsilon}=\int_{\Omega_{1}(x_{0})}(| \nabla_{\mathbb{B}}u_{\varepsilon}|^{2}+| \nabla_{\mathbb{B}}v_{\varepsilon}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\geq W_{0} ~~{\rm for}~~0<\varepsilon<\frac{p-1}{2}, \end{equation*} $

(3.17)

where $ W_{0}$ is a positive constant independent of $\varepsilon$. Then $u_{\varepsilon}, \nu_{\varepsilon}$ are uniformly bounded in $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))$. Passing to a subsequence, we may assume that $u_{\varepsilon}\rightharpoonup u_{0}, ~v_{\varepsilon}\rightharpoonup v_{0}$ weakly in $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))$ as $\varepsilon\rightarrow0$. Then $(u_{0}, v_{0})$ is a solution of the following problem

$ \begin{cases} -\Delta_{\mathbb{B}}u=\mu_{1}| u|^{2p-2}u+\beta| u|^{p-2}u|v|^{p}, x \in \Omega_{1}(x_{0}), \\ -\Delta_{\mathbb{B}}v=\mu_{2}| v|^{2p-2}v+\beta | v| ^{p-2}v | u| ^{p}, x \in \Omega_{1}(x_{0}) , \\ u, v \in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0})). \end{cases} $

Note that $u_{\varepsilon}(x_{0})=\max\limits_{\Omega_{1}(x_{0})}u_{\varepsilon}(x), v_{\varepsilon}(x_{0})=\max\limits_{\Omega_{1}(x_{0})}v_{\varepsilon}(x)$ and define $K_{\varepsilon}=\max\{u_{\varepsilon}(x_{0}), v_{\varepsilon}(x_{0})\}$. We claim that $K_{\varepsilon}\rightarrow\infty$ as $\varepsilon\rightarrow0$. Suppose the contrary. If $K_{\varepsilon}$ is uniformly bounded, then by the dominated convergent theorem, we have that

$ \begin{eqnarray*} &&\lim\limits_{\varepsilon\rightarrow0}\int_{\Omega_{1}(x_{0})} u_{\varepsilon}^{2p-2\varepsilon}\frac{dx_{1}}{x_{1}}dx^{'} =\int_{\Omega_{1}(x_{0})} u_{0}^{2p}\frac{dx_{1}}{x_{1}}dx^{'}, \\ &&\lim\limits_{\varepsilon\rightarrow0}\int_{\Omega_{1}(x_{0})} v _{\varepsilon}^{2p-2\varepsilon}\frac{dx_{1}}{x_{1}}dx^{'} =\int_{\Omega_{1}(x_{0})} v _{0}^{2p}\frac{dx_{1}}{x_{1}}dx^{'}, \\ &&\lim\limits_{\varepsilon\rightarrow0}\int_{\Omega_{1}(x_{0})} u_{\varepsilon}^{p-\varepsilon}v_{\varepsilon}^{p-\varepsilon}\frac{dx_{1}}{x_{1}}dx^{'} =\int_{\Omega_{1}(x_{0})} u_{0}^{p}v_{0}^{p}\frac{dx_{1}}{x_{1}}dx^{'}. \end{eqnarray*} $

Note that $E^{'}_{\varepsilon}(u_{\varepsilon}, v_{\varepsilon})=E^{'}(u_{0}, v_{0})=0$. It is standard to show that $u_{\varepsilon}^{*}\rightarrow u_{0}, v_{\varepsilon}^{*}\rightarrow v_{0}$ strongly in $\mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{1}(x_{0}))$ as $\varepsilon\rightarrow0$. By (3.17), we get that $(u_{0}, v_{0})\neq(0, 0)$. Moreover, $u_{0}\geq0, v_{0}\geq0$. By the strong maximum principle, $u_{0}>0, v_{0}>0$ in $\Omega_{1}(x_{0})$. Combining this with Pohozaev identity (see [11]), we get

$ \begin{equation*} 0 < \int_{\partial\Omega_{1}(x_{0})}(|\nabla_{\mathbb{B}}u_{0}|^{2}+| \nabla_{\mathbb{B}}v_{0}|^{2})\big[(\ln x_{1}, x^{'})\cdot\upsilon \big]dS=0, \end{equation*} $

which is a contradiction, here $\upsilon$ denotes the outward unit normal vector on $\partial\Omega_{1}(x_{0})$. So $K_{\varepsilon}\rightarrow+\infty$ as $\varepsilon\rightarrow0$. Define

$ \begin{equation*} U_{\varepsilon}(x_{1}, x^{'})=K_{\varepsilon}^{-1}u_{\varepsilon}(x_{1}^{K_{\varepsilon}^{-\alpha_{\varepsilon}}}, K_{\varepsilon}^{-\alpha_{\varepsilon}}x^{'}), ~~ V_{\varepsilon}(x_{1}, x^{'})=\\K_{\varepsilon}^{-1}v_{\varepsilon}(x_{1}^{K_{\varepsilon}^{-\alpha_{\varepsilon}}}, K_{\varepsilon}^{-\alpha_{\varepsilon}}x^{'}), ~~\alpha_{\varepsilon}=p-1-\varepsilon. \end{equation*} $

Then we have

$ \begin{equation*} 1=\max\{U_{\varepsilon}(x_{0}), V_{\varepsilon}(x_{0})\}=\\\max\{\max\limits_{x\in\Omega_{K_{\varepsilon}^{-\alpha_{\varepsilon}}} (x_{0})}U_{\varepsilon}(x), \max\limits_{x\in\Omega_{K_{\varepsilon}^{-\alpha_{\varepsilon}}} (x_{0})}V_{\varepsilon}(x)\}, \end{equation*} $

(3.18)

and $U_{\varepsilon}, V_{\varepsilon}$ satisfy

$ \begin{cases} -\Delta_{\mathbb{B}}U_{\varepsilon}=\mu_{1} U_{\varepsilon}^{2p-2\varepsilon-1}+\beta U_{\varepsilon}^{p-1-\varepsilon}V_{\varepsilon}^{p-\varepsilon}, x \in \Omega_{K_{\varepsilon}^{\alpha_{\varepsilon}}}(x_{0}), \\ -\Delta_{\mathbb{B}}V_{\varepsilon}=\mu_{2} V_{\varepsilon}^{2p-2\varepsilon-1}+\beta V_{\varepsilon}^{p-1-\varepsilon}U_{\varepsilon}^{p-\varepsilon}, x \in \Omega_{K_{\varepsilon}^{\alpha_{\varepsilon}}}(x_{0}), \\ U_{\varepsilon}, V_{\varepsilon} \in \mathcal{H}_{2, 0}^{1, \frac{N}{2}}(\Omega_{K_{\varepsilon}^{\alpha_{\varepsilon}}}(x_{0})). \end{cases} $

Since

$ \begin{equation*} \int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} U_{\varepsilon}|^{2}\frac{dx_{1}}{x_{1}}dx^{'} =K_{\varepsilon}^{-(N-2)\varepsilon}\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u_{\varepsilon}|^{2}\frac{dx_{1}}{x_{1}}dx^{'} \leq\int_{\mathbb{R}_{+}^{N}}|\nabla_{\mathbb{B}} u_{\varepsilon}|^{2}\frac{dx_{1}}{x_{1}}dx^{'}, \end{equation*} $

we get that $\{(U_{\varepsilon}, V_{\varepsilon})\}$ is bounded in $D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N})\times D_{2}^{1, \frac{N}{2}}(\mathbb{R}_{+}^{N})=D$. By elliptic estimates, up to a subsequence, we have $(U_{\varepsilon}, V_{\varepsilon})\rightarrow(U, V)\in D$ uniformly in every compact subset of $\mathbb{R}_{+}^{N}$ as $\varepsilon\rightarrow0$, and $(U, V)$ satisfies (1.6), that is $E^{'}(U, V)=0$. Moreover, $U, V\geq0$ are partly radially symmetric decreasing. Note that (3.18) we get $(U, V)\neq(0, 0)$, and so $(U, V)\in\mathcal{M}^{'}$. Then we deduce from (3.16) that

$ \begin{align*} B^{'}\leq E(U, V)&=(\frac{1}{2}-\frac{1}{2p})\int_{\mathbb{R}_{+}^{N}}(|\nabla_{\mathbb{B}} U|^{2}+|\nabla_{\mathbb{B}} V|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &\leq\liminf\limits_{\varepsilon\rightarrow0}(\frac{1}{2}-\frac{1}{2p-2\varepsilon}) \int_{\Omega_{K_{\varepsilon}^{\alpha_{\varepsilon}}}(x_{0})}(|\nabla_{\mathbb{B}} U_{\varepsilon}|^{2}+|\nabla_{\mathbb{B}} V_{\varepsilon}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &\leq\liminf\limits_{\varepsilon\rightarrow0}(\frac{1}{2}-\frac{1}{2p-2\varepsilon}) \int_{\Omega_{1}(x_{0})}(|\nabla_{\mathbb{B}} u_{\varepsilon}|^{2}+|\nabla_{\mathbb{B}} v_{\varepsilon}|^{2})\frac{dx_{1}}{x_{1}}dx^{'}\\ &=\liminf\limits_{\varepsilon\rightarrow0}B_{\varepsilon}\leq B^{'}. \end{align*} $

So $E(U, V)=B^{'}$. Note that $B^{'} < \min\{ \inf\limits_{(u, 0)\in \mathcal{M}^{'}}E(u, 0), \inf\limits_{(0, v)\in \mathcal{M}^{'}}E(0, v)\}$ and we have $U\not\equiv0, V\not\equiv0$. By the strong maximum principle, $U>0$, $V>0$ are partly radially symmetric decreasing. We also have $(U, V)\in\mathcal{M}$, and so $E(U, V)\geq B\geq B^{'}$, that is, $E(U, V)=B=B^{'}.$ Moreover $(U, V)$ is positive least energy solution of (1.6), which is partly radially symmetric decreasing.

Finally, with the help of (2.1) and [1, (2-2) in Theorem 1.6], we get that there exists $d(\beta)$ and $g(\beta)$ on $(-\beta_{2}, \beta_{2})$ for some $\beta_{2}>0$, and $l_{i}(d(\beta), g(\beta))\equiv0$ for $i=1, 2$. This implies that $(\sqrt{d(\beta)}U_{\varepsilon}, \sqrt{g(\beta)}U_{\varepsilon})$ is a positive solution of (1.6). Therefore we have

$ \begin{equation*} \lim\limits_{\beta\rightarrow0}(d(\beta)+g(\beta))=d(0)+g(0)=\mu_{1}^{-\frac{N-2}{2}}+\mu_{2}^{-\frac{N-2}{2}}, \end{equation*} $

that is, there exists $0<-\beta_{1} \leq-\beta_{2}$ such that

$ \begin{equation*} d(\beta)+g(\beta)>\min\{\mu_{1}^{-\frac{N-2}{2}}, \mu_{2}^{-\frac{N-2}{2}}\} ~~{\rm for~~ any}~~\beta\in(0, \beta_{1}). \end{equation*} $

Recall that

$ \begin{eqnarray*} B\leq E(\sqrt{d_{0}}U_{\varepsilon}, \sqrt{g_{0}}U_{\varepsilon})=\frac{1}{N}(d_{0}+g_{0})S^{\frac{N}{2}}, ~~ B^{'}<\min\{\frac{1}{N}\mu_{1}^{-\frac{N-2}{2}}S^{\frac{N}{2}}, \frac{1}{N}\mu^{-\frac{N-2}{2}}_{2}S^{\frac{N}{2}}\}, \end{eqnarray*} $

and we have

$ \begin{equation*} E(U, V)=B^{'}=B<E(\sqrt{d(\beta)}U_{\varepsilon}, \sqrt{g(\beta)}U_{\varepsilon})~~{\rm for}~~\beta\in(0, \beta_{1}), \end{equation*} $

that is, $(\sqrt{d(\beta)}U_{\varepsilon}, \sqrt{g(\beta)}U_{\varepsilon})$ is a different positive solution of (1.6) with respect to $(U, V)$. We complete the proof of (2-2) in Theorem 1.2.