All groups in this paper are finite and our notation is standard (see [1]). Let $\Sigma$ be an abstract group theoretical property, for example, nilpotency, supersolvability, solvability, etc. If all proper subgroups of a group $G$ have the property $\Sigma$ but $G$ does not have the property $\Sigma$, then $G$ is called a minimal non- $\Sigma$-group.

One of the hottest topics in group theory is to determinate the structure of minimal non- $\Sigma$-groups and many meaningful results about this topic were obtained. The specific papers about this topic can refer to [2-10].

The aim of this paper is to study the structure of a kind of minimal non- $\Sigma$-groups. We call the groups whose 2-maximal subgroups are subnormal SMSN-groups. A group $G$ is a minimal non-SMSN-group if every proper subgroup of $G$ is an SMSN-group but $G$ itself is not, and we classify the minimal non-SMSN-groups completely.

In this section, we give some definitions and some lemmas needed in this paper.

Lemma 2.1(see [5, Lemma 5])  Every 2-maximal subgroup of a group $G$ is subnormal if and only if either $G$ is nilpotent or $G$ is a Schmidt group with abelian Sylow subgroups.

Lemma 2.2  If $G$ is a solvable minimal non-SMSN-group, then $|\pi(G)|\leq 3$.

Proof  If $|\pi(G)|\geq 4$, then every maximal subgroup of $G$ has at least three prime divisors since $G$ is solvable. Applying Lemma 2.1, $G$ is minimal non-nilpotent, a contradiction. Hence $|\pi(G)|\leq 3$.

Lemma 2.3(see [10])  Any minimal simple group (non-abelian simple group all of whose proper subgroups are solvable) is isomorphic to one of the following simple groups

(1) PSL(3, 3);

(2) PSL$(2, p)$, where $p$ is a prime with $p>3$ and $5\nmid p^{2}-1$;

(3) PSL$(2, 2^{q})$, where $q$ is a prime;

(4) PSL$(2, 3^{q})$, where $q$ is an odd prime;

(5) The Suzuki group Sz$(2^{q})$, where $q$ is an odd prime.

Lemma 2.4(see [11])  Suppose that $p'$-group $H$ acts on a $p$-group $G$. Let

$ \begin{equation*} \Omega(G)= \begin{cases} \Omega_{1}(G),&p>2, \\\Omega_{2}(G),&p=2. \end{cases} \end{equation*} $

If $H$ acts trivially on $\Omega(G)$, then $H$ acts trivially on $G$ as well.

Lemma 2.5(see [7, Lemma 2.9])  If a $p$-group $G$ of order $p^{n+1}$ has a unique non-cyclic maximal subgroup, then $G$ is isomorphic to one of the following groups

(Ⅰ) $C_{p^{n}}\times C_p=\langle a, b\mid a^{p^n}=b^p=1, [a, b]=1\rangle $, where $n\geq 2$;

(Ⅱ) $M_{p^{n+1}}=\langle a, b\mid a^{p^n}=b^p=1, b^{-1}ab=a^{1+p^{n-1}}\rangle$, where $n\geq 2$ and $n\geq 3$ if $p=2$.

Lemma 2.6(see [12])  Let $G$ be a group and $H$ a nilpotent subnormal subgroup of $G$. Then $G$ contains a nilpotent normal subgroup of $G$ containing $H$.

In this section, we give the specific classification of the minimal non-SMSN-groups.

Theorem 3.1  A non-solvable group $G$ is a minimal non-SMSN-group if and only if $G$ is isomorphic to $A_{5}$, where $A_{5}$ is the alternating group of degree 5.

Proof  We only prove the necessity part.

Since $G$ is a non-solvable group whose maximal subgroups are all SMSN-groups, then $G$ is a minimal non-solvable group by Lemma 2.1, and so $G/\Phi(G)$ is a minimal simple group.

Case 1  Assume $\Phi(G)=1$. Then $G$ is isomorphic to one of the simple groups mentioned in Lemma 2.3.

Let $G\cong $ PSL(3, 3). Then $G$ has a subgroup which is isomorphic to $S_{4}$ by [13, p.13], but $S_{4}$ is not an SMSN-group, a contradiction. So $G\ncong $ PSL(3, 3).

Let $G\cong $ PSL $(2, p)$, where $p$ is a prime with $p>3$ and $5\nmid p^{2}-1$. If $p\geq 13$, then there exists a maximal subgroup of $G$ which is isomorphic to a dihedral group $D_{p-1}$ or $D_{p+1}$ by [14, Corollary 2.2]. Certainly, 4 divides the order of either $D_{p-1}$ or $D_{p+1}$, say $A$, and $A$ is not an SMSN-group applying Lemma 2.1, a contradiction. If $p=7$, then $p^{2}\equiv 1$ (mod 16). By [14, Corollary 2.2], $G$ has a subgroup which is isomorphic to $S_{4}$, but $S_{4}$ is not an SMSN-group, a contradiction. Hence $p=5$ and $G\cong A_{5}$.

Let $G\cong $ PSL $(2, 2^{q})$. By [14, Corollary 2.2], $G$ has maximal subgroups: the dihedral groups of order $2(2^{q}\pm 1)$; the Frobenius group $H$ of order $2^{q}(2^{q}-1)$; the alternating group $A_{4}$ of degree $4$ when $q=2$. Clearly, $G\cong A_{5}$ when $q=2$ and it is a minimal non-SMSN-group. If $q>2$, then $3\mid 2^{q}+1$. It follows from Lemma 2.1 that $G$ is not a minimal non-SMSN-group.

Let $G\cong $ PSL $(2, 3^{q})$. Similar arguments as above, $G$ has a dihedral group $B$ whose Sylow 2-subgroups are neither cyclic nor normal, which contradicts the fact that $B$ is an SMSN-group. So $G\ncong $ PSL $(2, 3^{q})$.

Let $G\cong $ Sz $(2^{q})$. By [15, Theorem 9], $G$ has a Frobenius group $K$ of order $4(2^{q}\pm 2^{\frac{q+1}{2}}+1)$, but the Sylow 2-subgroups of $K$ are neither cyclic nor normal, a contradiction. So $G\ncong $ Sz$(2^{q})$.

Case 2  Assume $\Phi(G)\neq 1$. It is easy to see that $\Phi(G/\Phi(G))=1$ and $G/\Phi(G)$ is a non-solvable minimal non-SMSN-group. Similar arguments as above and by induction, $G/\Phi(G)\cong A_{5}$. Hence $G$ has two non-nilpotent maximal subgroups $M_{1}$ and $M_{2}$ such that $M_{1}/\Phi(G)\cong A_{4}$ and $M_{2}/\Phi(G)\cong D_{10}$, where $A_{4}$ is the alternating group of degree 4 and $D_{10}$ is the dihedral group of order 10. Since $M_{1}$ and $M_{2}$ are SMSN-groups, they are minimal non-nilpotent by Lemma 2.1. It makes $|G|=2^{a}\cdot 3\cdot 5$ and $|\Phi(G)|=2^{a-2}$, where $a\geq 3$. By Lemma 2.1 again, the Sylow 2-subgroups of $M_{1}$ are elementary abelian. At the same time, the Sylow 2-subgroups of $M_{2}$ are cyclic whose orders are more than 2 by Lemma 2.1, a contradiction.

Theorem 3.2  The minimal non-SMSN-group $G$ whose order has exactly two prime divisors $p$ and $q$ is exactly one of the following types ( $P$ and $Q$ are Sylow subgroups)

(1) $G=\langle x, y \mid x^{p}=y^{q^{n}}=1, y^{-1}xy=x^{i}\rangle $, where $i^{q}\not\equiv 1($mod $\ p)$, $i^{q^{2}}\equiv 1($mod $\ p)$, $p>q$, $n\geq 2$ and $0 < i < p$;

(2) $G=\langle x, y\mid x^{pq}=y^{q}=1, y^{-1}xy=x^{i}\rangle $, where $p\equiv 1($mod $\ q)$, $i\equiv 1($mod $\ q)$, $i^{q}\equiv 1($mod $\ p)$ and $1 < i < p$;

(3) $G=\langle x, y\mid x^{4p}=1, y^{2}=x^{2p}, y^{-1}xy=x^{-1}\rangle $;

(4) $G=\langle x, y, z\mid x^{p}=y^{q^{n-1}}=z^{q}=1, y^{-1}xy=x^{i}, [x, z]=1, [y, z]=1\rangle $ where $p>q$, $i\not\equiv 1($mod $\ p)$, $i^{q}\equiv 1($mod $\ p)$ and $n\geq 3$;

(5) $G=\langle x, y, z\mid x^{p}=y^{q^{n-1}}=z^{q}=1, y^{-1}xy=x^{i}, [x, z]=1, z^{-1}yz=y^{1+q^{n-2}}\rangle $, where $p>q$, $i\not\equiv 1($mod $\ p)$, $i^{q}\equiv 1($mod $\ p)$, $n\geq 3$ and $n\geq 4$ if $q=2$;

(6) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle y\rangle $ with $|y|=q^{n}$ and $n\geq 2$, $\langle y^{q}\rangle $ acts irreducibly on $P$ and $\langle y^{q^{2}}\rangle $ centralizes $P$;

(7) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group and $r\geq 2$, $Q=\langle y\rangle $ with $|y|=q^{n}$ and $n\geq 1$, $[a_{1}, Q]=1$, $Q$ acts irreducibly on $\langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ and $\Phi(Q)$ centralizes $P$;

(8) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle y\rangle $ with $|y|=q^{n}$ and $n\geq 1$, $Q$ acts irreducibly on $\langle a_{1}\rangle \times \cdots \times \langle a_{l-1}\rangle $ and $\langle a_{l}\rangle \times \cdots \times \langle a_{r}\rangle $ with $l\geq 2$, and $\Phi(Q)$ centralizes $P$;

(9) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ $(r\geq 2)$ is a $p$-group with $|a_{1}|=|a_{2}|=\cdots =|a_{r}|=p^{2}$, $Q=\langle y\rangle $ with $|y|=q^{n}$ and $n\geq 1$, $Q$ acts irreducibly on $\Phi(P)$, $\Phi(Q)$ centralizes $P$, and $G/\Phi(P)$ is a minimal non-abelian group;

(10) $G=P\rtimes Q$, where $P$ is a non-abelian special $p$-group of rank $2m$, the order of $p$ modulo $q$ being $2m$, $Q=\langle y\rangle $ is cyclic of order $q^{r}>1$, $y$ induces an automorphism in $P$ such that $P/\Phi(P)$ is a faithful and irreducible $Q$-module, and $y$ centralizes $\Phi(P)$. Furthermore, $|P/\Phi(P)|=p^{2m}$ and $|P'|\leq p^{m}$;

(11) $G=P\rtimes Q$, where $P$ is a non-abelian special $p$-group with exp $(P)\leq p^{2}$ and $|\Phi(P)|\geq p^{2}$, $Q=\langle y\rangle $ with $|y|=q^{n}$ and $n\geq 1$, $Q$ acts irreducibly on $\Phi(P)$, $\Phi(Q)$ centralizes $P$, and $G/\Phi(P)$ is a minimal non-abelian group;

(12) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle a, b\mid a^{q}=b^{q}=1$, $[a, b]=1\rangle, [P, b]=1$, $\langle a\rangle $ acts irreducibly on $P$;

(13) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle a, b\mid a^{q}=b^{q}=1, [a, b]=1\rangle $, $\langle a\rangle $ and $\langle b\rangle $ act irreducibly on $P$;

(14) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle a, b\mid a^{4}=1, b^{2}=a^{2}, b^{-1}ab=a^{-1}\rangle, [P, b]=1$ and $\langle a\rangle $ acts irreducibly on $P$;

(15) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle a, b\mid a^{4}=1, b^{2}=a^{2}, b^{-1}ab=a^{-1}\rangle $, $[P, a^{2}]=1$, $\langle a\rangle $ and $\langle b\rangle $ act irreducibly on $P$;

(16) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle y, z\mid y^{q^{n-1}}=z^q=1, [y, z]=1\rangle $ with $n\geq 3$, $z\in Z(G)$, $\langle y\rangle $ acts irreducibly on $P$ and $\langle y^{q}\rangle $ centralizes $P$;

(17) $G=P\rtimes Q$, where $P=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r}\rangle $ is an elementary abelian $p$-group with $r\geq 2$, $Q=\langle y, z\mid y^{q^{n-1}}=z^q=1, z^{-1}yz=y^{1+q^{n-2}}\rangle$ with $n\geq 3$ and $n\geq 4$ if $q=2$, $\langle z\rangle \leq C_{G}(P)$, $\langle y\rangle $ acts irreducibly on $P$ and $\langle y^{q}\rangle $ centralizes $P$;

(18) $G=PQ$, where $P=\langle x\rangle \ntrianglelefteq G$ with $|P|=p$, $Q=(\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r-1}\rangle )\rtimes \langle a_{r}\rangle $ is a non-normal $q$-group with $r\geq 3$, $F(G)=O_{q}(G)=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r-1}\rangle $, $P$ acts irreducibly on $O_{q}(G)$, $a_{r}^{-1}xa_{r}=x^{i}$ and $p>q$, where $i$ is a primitive $q$-th root of unity modulo $p$, $F(G)$ is the Fitting subgroup of $G$.

Proof  If $G$ is a solvable minimal non-SMSN-group whose order has exactly two prime divisors, then we assume $G=PQ$, where $P\in $Syl$_{p}(G)$ and $Q\in $Syl$_{q}(G)$.

Assume that $P$ and $Q$ are neither cyclic nor normal in $G$. The solvability of $G$ implies that $G$ has a normal subgroup $M$ of prime index, say $q$. Let $M_{p}$ be a Sylow $p$-subgroup of $M$. Since $M$ is an SMSN-group, we have that $M_{p}$ is either cyclic or normal in $M$ by Lemma 2.1. Clearly $M_{p}$ must be normal in $M$ since it is also a Sylow $p$-group of $G$. Now it follows from $M_{p}$ char $M\unlhd G$ that $M_{p}\unlhd G$, a contradiction. So $G$ has a Sylow subgroup which is either cyclic or normal.

(1) Assume that $P$ and $Q$ are cyclic and let $P=\langle x\rangle $ and $Q=\langle y\rangle $ with $|x|=p^{m}$, $|y|=q^{n}$ and $p>q$. In this case, $y^{-1}xy=x^{i}$ with $i^{q^{n}}\equiv 1($mod $\ p^{m})$, $0 < i < p^{m}$ and $(p^{m}, q^{n}(i-1))=1$. Considering the maximal subgroups $P\langle y^{q}\rangle $ and $\langle x^{p}\rangle Q$ of $G$, if $\langle x^{p}\rangle Q=\langle x^{p}\rangle \times Q$, then by Lemma 2.4, $G$ is nilpotent, a contradiction. This implies $\langle x^{p}\rangle Q=\langle x^{p}\rangle \rtimes Q$. By Lemma 2.1, $x^{p}=1$, $\langle y^{q}\rangle $ is not normal in $G$, but $\langle y^{q^{2}}\rangle $ is normal in $G$. So $i^{q}\not\equiv 1($mod $\ p)$, $i^{q^{2}}\equiv 1($mod $\ p)$ and $G$ is of type (1).

(2) Assume that $P$ is a cyclic normal subgroup of $G$ and $Q$ is neither cyclic nor normal in $G$. If $q>p$, then by Burnside's theorem [1, 10.1.8], $Q\unlhd G$, a contradiction. So $q<p$. If $Q$ has two non-cyclic maximal subgroups $Q_{1}$ and $Q_{2}$, then by Lemma 2.1, $PQ_{1}=P\times Q_{1}$, $PQ_{2}=P\times Q_{2}$ and so $Q=Q_{1}Q_{2}$ is normal in $G$, a contradiction. Therefore, every maximal subgroup of $Q$ is cyclic or $Q$ has a unique non-cyclic maximal subgroup, and so $Q$ is an elementary abelian $q$-group of order $q^{2}$, the quaternion group $Q_{8}$ or one of the types in Lemma 2.5.

Case 1  Assume $P=\langle z\rangle $ and $Q=\langle a, b\mid a^{q}=b^{q}=1, [a, b]=1\rangle $. If $\langle a\rangle $ and $\langle b\rangle $ acting on $P$ by conjugation are both trivial, then $G$ is nilpotent, a contradiction. Therefore, we may assume that $\langle a\rangle $ acting on $P$ by conjugation is non-trivial. By Lemma 2.1, $z^{p}=1$. If $C_{G}(P)=P$, then $G/C_{G}(P)$ is an elementary abelian $q$-group of order $q^{2}$. However, $G/C_{G}(P)\lesssim $Aut $(P)$, and Aut $(P)$ is cyclic, a contradiction. Hence $b$ is contained in $C_{G}(P)$. Clearly, $C_{G}(P)=\langle x\rangle $, $y^{-1}xy=x^{i}$, $|x|=pq$, $y=a$, $q|p-1$, $i\equiv 1($mod $\ q)$ and $i^{q}\equiv 1($mod $\ p)$, where $x=zb$ is a generator of $C_{G}(P)$. So $G$ is of type (2).

Case 2  Assume $P=\langle z\rangle $ and $Q=Q_{8}=\langle a, b\mid a^{4}=1, b^{2}=a^{2}, b^{-1}ab=a^{-1}\rangle $. Similar arguments as Case 1, we have that $z^{p}=1$, $b$ is contained in $C_{G}(P)$ and $|Z(G)|=2$. So $C_{G}(P)=\langle x\rangle $ with $|x|=4p$, $y=a$, $y^{-1}xy=x^{i}$ and $i^{2}\equiv 1($mod $\ 4p)$, where $x=zb$ is a generator of $C_{G}(P)$. By computations, $G$ is of type (3).

Case 3  Assume that $P=\langle x\rangle $ and $Q$ is the type of Lemma 2.5 (Ⅰ) with $|Q|=q^{n}$. Namely, $Q=\langle y, z\mid y^{q^{n-1}}=z^q=1, [y, z]=1\rangle $, where $n\geq 3$. Then $Q$ has maximal subgroups $H=\langle y\rangle $, $K_0=\langle y^q, z\rangle $ and $K_{s}=\langle y^q, zy^s\rangle=\langle zy^s\rangle $ with $s=1, \cdots, q-1$, where $K_0$ is the unique non-cyclic maximal subgroup of $Q$. By hypothesis and Lemma 2.1, $PH\neq P\times H$, $PK_{0}=P\times K_{0}$ and $x^{p}=1$. Hence $G=\langle x, y, z\mid x^{p}=y^{q^{n-1}}=z^{q}=1, y^{-1}xy=x^{i}, [x, z]=1, [y, z]=1\rangle $, where $i\not\equiv 1($mod $\ p)$, $i^{q}\equiv 1($mod $\ p)$. So $G$ is of type (4).

Case 4  Assume that $P=\langle x\rangle $ and $Q$ is the type of Lemma 2.5 (Ⅱ) with $|Q|=q^{n}$. Namely, $Q=\langle y, z\mid y^{q^{n-1}}=z^q=1, z^{-1}yz=y^{1+q^{n-2}}\rangle $, where $n\geq 3$ and $n\geq 4$ if $q=2$. In the similar way as above, we have that $x^{p}=1$, $\langle z\rangle \leq C_{G}(P)$ and $y^{-1}xy=x^{i}$, where $i\not\equiv 1($mod $\ p)$ and $i^{q}\equiv 1($mod $\ p)$. So $G$ is of type (5).

(3) Assume that $P$ is a non-cyclic normal subgroup of $G$ and $Q=\langle y\rangle $ is non-normal cyclic subgroup of $G$ with $|y|=q^{n}$. If there exists a subgroup $P^{*}$ of $P$ with $1<\Phi(P)<P^{*}<P$ such that $P^{*}Q=QP^{*}$, then $P^{*}\unlhd G$ since $P^{*}$ is subnormal in $G$. By Maschke's theorem [1, 8.1.2], $P$ has a subgroup $K$ with $1<K<P$ such that $P/\Phi(P)=P^{*}/\Phi(P)\times K/\Phi(P)$, $K\unlhd G$, $K\neq P^{*}$, and at least one of $P^{*}Q$ and $KQ$ is a non-nilpotent SMSN-group. By Lemma 2.1, it is easy to see that $P^{*}\cap K=\Phi(P)=1$, a contradiction. Hence $\Phi(P)=1$ or $P/\Phi(P)$ is the minimal normal subgroup of $G/\Phi(P)$ when $\Phi(P)\neq 1$.

Case 1  Assume $\Phi(P)=1$. If $P$ is a minimal normal subgroup of $G$, then by hypothesis, the maximal subgroup $P\Phi(Q)$ of $G$ is non-nilpotent. By Lemma 2.1, $\langle y^{q}\rangle $ acts irreducibly on $P$ and $[P, y^{q^{2}}]=1$. So $G$ is of type (6). If $P$ has a non-trivial proper subgroup $P_{1}$ which is normal in $G$, then there exists a subgroup $P_{2}$ of $P$ such that $P=P_{1}\times P_{2}$ and $P_{2}\unlhd G$ by Maschke's theorem [1, 8.1.2]. Clearly, at least one action that $\langle y\rangle $ acts on $P_{1}$ and $P_{2}$ by conjugation is non-trivial. If $P_{1}Q=P_{1}\times Q$ and $P_{2}Q=P_{2}\rtimes Q$, then by Maschke's theorem [1, 8.1.2] and Lemma 2.1, it is easy to see that $|P_{1}|=p$, $[P, y^{q}]=1$ and $G$ is of type (7). If $P_{1}Q=P_{1}\rtimes Q$ and $P_{2}Q=P_{2}\rtimes Q$, then by Lemma 2.1, $\langle y\rangle $ acts irreducibly on $P_{1}$ and $P_{2}$, and $[P, y^{q}]=1$. So $G$ is of type (8).

Case 2  Assume $\Phi(P)>1$ and $Z(P)=P$. By the same arguments as the beginning of (3), it is easy to see that $\Phi(P)$ is the unique normal subgroup of $G$ which is contained in $P$, and so $P$ is a homocyclic $p$-group (a product of some cyclic subgroups of the same order). By Lemma 2.1 and Lemma 2.4, we have easily that the exponent of $P$ is $p^{2}$, one maximal subgroup $P\Phi(Q)$ of $G$ is nilpotent. Hence another maximal subgroup $\Phi(P)Q$ is non-nilpotent, and $\langle y\rangle $ acts irreducibly on $\Phi(P)$. Clearly the quotient group $G/\Phi(P)$ is a minimal non-abelian group. So $G$ is of type (9).

Case 3  Assume $\Phi(P)>1$ and $Z(P)<P$. Similarly, $\Phi(P)=Z(P)=P'$ is the unique non-trivial characteristic subgroup of $P$, that is, $P$ is a special $p$-group with exp $(P)\leq p^{2}$ and $P\Phi(Q)$ is nilpotent. If $\Phi(P)Q$ is nilpotent also, then by a result in [4, Theorem 2], $G$ is of type (10). If $|\Phi(P)|=p$ and $p<q$, then $G$ belongs to type (10). If $\Phi(P)Q$ is non-nilpotent with $|\Phi(P)|=p$ and $p>q$, then $G$ is minimal non-supersolvable. Examining a result in [4, Theorem 10], $G$ is not isomorphic to anyone of them. If $\Phi(P)Q$ is non-nilpotent with $|\Phi(P)|\geq p^{2}$, then the quotient group $G/\Phi(P)$ is a minimal non-abelian group. So $G$ is of type (11).

(4) Assume that $P$ is a non-cyclic normal subgroup of $G$ and $Q$ is neither cyclic nor normal in $G$. If $\Phi(P)>1$, then by Lemma 2.1, $PQ_{1}$ and $PQ_{2}$ are both nilpotent and so $G$ is nilpotent, a contradiction, where $Q_{1}$ and $Q_{2}$ are two distinct maximal subgroups of $Q$. Hence $P$ is an elementary abelian $p$-group of order $p^{r}$ with $r\geq 2$. Similar arguments as in (2), $Q$ is an elementary abelian $q$-group of order $q^{2}$, the quaternion group $Q_{8}$ or one of the types in Lemma 2.5.

Case 1  Let $Q=\langle a, b\mid a^{q}=b^{q}=1, [a, b]=1\rangle $. Clearly, there exists a non-trivial automorphism that $\langle a\rangle $ or $\langle b\rangle $ acts on $P$ by conjugation. We may assume that $\langle a\rangle $ acting on $P$ by conjugation is non-trivial and $\langle b\rangle $ acting on $P$ by conjugation is trivial. So $G$ is of type (12). If $\langle a\rangle $ and $\langle b\rangle $ acting on $P$ by conjugation are both non-trivial, then $G$ is of type (13).

Case 2  Let $Q=Q_{8}=\langle a, b\mid a^{4}=1, b^{2}=a^{2}, b^{-1}ab=a^{-1}\rangle $. Similar arguments as above, $G$ is of either type (14) or type (15).

Case 3  Let $Q$ be as in Lemma 2.5 (Ⅰ) with $|Q|=q^{n}$. Namely, $Q=\langle y, z\mid y^{q^{n-1}}=z^q=1, [y, z]=1\rangle $, where $n\geq 3$. Similar arguments as Case 3 in (2), $\langle y\rangle $ acts irreducibly on $P$, $[P, y^{q}]=1$ and $z\in Z(G)$. So $G$ is of type (16).

Case 4  Let $Q$ be as in Lemma 2.5 (Ⅱ) with $|Q|=q^{n}$. Namely, $Q=\langle y, z\mid y^{q^{n-1}}=z^q=1, z^{-1}yz=y^{1+p^{n-2}}\rangle$, where $n\geq 3$ and $n\geq 4$ if $p=2$. Similar arguments as Case 4 in (2), $\langle y\rangle $ acts irreducibly on $P$, $[P, y^{q}]=1$ and $\langle z\rangle \leq C_{G}(P)$. So $G$ is of type (17).

(5) Assume that $P=\langle x\rangle $ is a non-normal cyclic subgroup of $G$ and $Q$ is neither cyclic nor normal in $G$. Clearly $p>q$. The solvability of $G$ implies that $G$ has a normal subgroup $M$ of prime index. If $|G:M|=p$, then it is easy to see that $G$ has a normal Sylow $q$-group since $M$ is an SMSN-group and applying Lemma 2.1, a contradiction. Therefore, $|G:M|=q$. If there exists a cyclic Sylow $q$-subgroup $M_{q}$ of $M$, then $M$ has a normal Sylow $p$-subgroup $M_{p}$, and so $M_{p}$ is a normal Sylow $p$-subgroup of $G$, a contradiction. Hence $M_{q}$ is non-cyclic and $|Q|\geq q^{3}$. By Lemma 2.1, $M_{q}$ is normal in $M$ and $M_{p}$ has a maximal subgroup $P_{1}$ such that $P_{1}$ is normal in $M$, where $M_{p}$ is a Sylow $p$-subgroup of $M$. Hence $M_{q}$ and $P_{1}$ are both subnormal in $G$. By Lemma 2.6, $F(G)=P_{1}\times M_{q}=O_{p}(G)\times O_{q}(G)$. Clearly, $O_{p}(G)=\langle x^{p}\rangle $ and $O_{q}(G)=\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r-1}\rangle $ is an elementary abelian $q$-group with $|O_{q}(G)|\geq q^{2}$. If $N_{G}(P)$ is nilpotent, then $N_{G}(P)=C_{G}(P)$ since $P$ is cyclic. By Burnside Theorem [1, 10.1.8], $G$ is $p$-nilpotent, a contradiction. Hence $N_{G}(P)=P\langle a_{r}\rangle $ is a Schmidt subgroup of $G$, and so $P\langle a_{r}^{q}\rangle $ is nilpotent with $|P|=p$ by Lemma 2.1, where $a_{r}$ is a $q$-element. Since $M$ is a Schmidt subgroup of $G$ also, $O_{q}(G)$ is a minimal normal subgroup of $G$ and $G=MN_{G}(P)$. Hence $O_{q}(G)\langle a_{r}\rangle $ is a Sylow $q$-subgroup of $G$ and $O_{q}(G)\cap \langle a_{r}\rangle =\langle a_{r}^{q}\rangle $. Furthermore, $|a_{r}|=q$ since $O_{q}(G)P$ is a Schmidt subgroup of $G$. If $\Phi(Q)=1$, then $Q$ is abelian. Hence $N_{G}(Q)=C_{G}(Q)=Q\lessdot G$. So $G$ is $q$-nilpotent, a contradiction. If $\Phi(Q)\neq 1$, then $Q=(\langle a_{1}\rangle \times \langle a_{2}\rangle \times \cdots \times \langle a_{r-1}\rangle )\rtimes \langle a_{r}\rangle $. So $G$ is of type (18).

Conversely, it is clear that the groups of types (1)-(18) are minimal non-SMSN-groups.

Theorem 3.3  The solvable minimal non-SMSN-group $G$ whose order has exactly three prime divisors $p, q$ and $r$ is exactly one of the following types ( $P, Q$ and $R$ are Sylow subgroups)

(1) $G=(P\times Q)\rtimes R$, where $[P, R]=1$, $|P|=p$, $Q$ is an elementary abelian $q$-group, $R=\langle a\rangle $ is cyclic, $R$ acts irreducibly on $Q$ and $\langle a^{r}\rangle $ centralizes $Q$;

(2) $G=(P\times Q)\rtimes R$, where $P$ and $Q$ are both elementary abelian, $R=\langle a\rangle $ is cyclic, $R$ acts irreducibly on $P$ and $Q$, $\langle a^{r}\rangle $ centralizes $PQ$;

(3) $G=P\rtimes (Q\times R)$, where $P$ is an elementary abelian $p$-group, $Q$ is a group of order $q$, $R$ is a group of order $r$, $Q$ and $R$ act irreducibly on $P$, respectively.

Proof  It is easy to see that $G$ has at least one normal Sylow subgroup and we assume that $G=PQR$, where $P\in $Syl$_{p}(G)$, $Q\in $Syl$_{q}(G)$, $R\in $Syl$_{r}(G)$, and $P\unlhd G, R\ntrianglelefteq G$. Clearly, we only need consider the following cases.

Case 1  If $Q\unlhd G, PR=P\times R$ and $QR=Q\rtimes R$, then $QR$ is an SMSN-group. By Lemma 2.1, $Q$ is an elementary abelian $q$-group and $R=\langle a\rangle $ is cyclic. If $|P|\neq p$, then $P_{1}QR$ is nilpotent by Lemma 2.1 again, a contradiction, where $1<P_{1}<P$. Hence $|P|=p$ and $G$ is of type (1).

Similarly, if $Q\unlhd G, PR=P\rtimes R$ and $QR=Q\times R$, then $G$ is isomorphic to type (1) also. If $Q\unlhd G, PR=P\rtimes R$ and $QR=Q\rtimes R$, it is easy to see that $G$ is of type (2).

Case 2  If $Q\ntrianglelefteq G$, $PQ=P\rtimes Q$, $PR=P\times R$, $QR=Q\rtimes R$, then by Lemma 2.1, $P$ is an elementary abelian $p$-group, $Q=\langle a\rangle $ is a cyclic group of order $q$, $q>r$ and $R=\langle b\rangle $ is cyclic. Since $C_{G}(P)=P\times R\unlhd N_{G}(P)=G$, $R\unlhd G$, a contradiction.

If $Q\ntrianglelefteq G$, $PQ=P\rtimes Q$, $PR=P\rtimes R$, $QR=Q\times R$, and $\Phi(R)\neq 1$, then $PQ\Phi(R)$ is nilpotent by Lemma 2.1, a contradiction. Hence $\Phi(R)=1$, then $G$ is of type (3).

Similarly, if $Q\ntrianglelefteq G$, $PQ=P\rtimes Q$, $PR=P\rtimes R$, $QR=Q\rtimes R$, then $P$ is elementary abelian, $Q=\langle a\rangle $ is a group of order $q$, $R=\langle b\rangle $ is a group of order $r$ and $r|q-1$. Let $|P|=p^{\alpha}, \alpha\geq 1$. Then by [16, Theorem 1.5], $p^{\alpha}\equiv 1($mod $\ q)$, $p^{\alpha}\equiv 1($mod $\ r)$. Hence $p^{\alpha}-1=qm=rn$, where $m$ and $n$ are integers. So $q=rnm^{-1}$, a contradiction.

Conversely, it is clear that the groups of types (1)-(3) are minimal non-SMSN-groups.

By Lemma 2.1, combining Theorem 3.1, Theorem 3.2 and Theorem 3.3, the complete classification of the minimal non-SMSN-groups is as follows.

Corollary 3.4  The minimal non-SMSN-groups are exactly the groups of $A_{5}$, types (1) to (18) of Theorem 3.2 and types (1) to (3) of Theorem 3.3, where $A_{5}$ is the alternating group of degree 5.