Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks

Introduction

Deep neural networks have achieved great success in many applications like image processing (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012) and Go games (Silver et al., 2016). However, the reason why deep networks work well in these fields remains a mystery for long time. Different lines of research try to understand the mechanism of deep neural networks from different aspects. For example, a series of work tries to understand how the expressive power of deep neural networks are related to their architecture, including the width of each layer and depth of the network (Telgarsky, 2015, 2016; Lu et al., 2017; Liang and Srikant, 2016; Yarotsky, 2017, 2018; Hanin, 2017; Hanin and Sellke, 2017). These work shows that multi-layer networks with wide layers can approximate arbitrary continuous function.

In this paper, we mainly focus on the optimization perspective of deep neural networks. It is well known that without any additional assumption, even training a shallow neural network is an NP-hard problem (Blum and Rivest, 1989). Researchers have made various assumptions to get a better theoretical understanding of training neural networks, such as Gaussian input assumption (Brutzkus et al., 2017; Du et al., 2017a; Zhong et al., 2017) and independent activation assumption (Choromanska et al., 2015; Kawaguchi, 2016). A recent line of work tries to understand the optimization process of training deep neural networks from two aspects: over-parameterization and random weight initialization. It has been observed that over-parameterization and proper random initialization can help the optimization in training neural networks, and various theoretical results have been established (Safran and Shamir, 2017; Du and Lee, 2018; Arora et al., 2018a; Allen-Zhu et al., 2018c; Du et al., 2018b; Li and Liang, 2018). More specifically, Safran and Shamir (2017) showed that over-parameterization can help reduce the spurious local minima in one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function. Du and Lee (2018) showed that with over-parameterization, all local minima in one-hidden-layer networks with quardratic activation function are global minima. Arora et al. (2018b) showed that over-parameterization introduced by depth can accelerate the training process using gradient descent (GD). Allen-Zhu et al. (2018c) showed that with over-parameterization and random weight initialization, both gradient descent and stochastic gradient descent (SGD) can find the global minima of recurrent neural networks.

In this paper, we aim to advance this line of research by studying the optimization properties of gradient-based methods for deep ReLU neural networks. In specific, we consider an $L$-hidden-layer fully-connected neural network with ReLU activation function. Similar to the one-hidden-layer case studied in Li and Liang (2018) and Du et al. (2018b), we study binary classification problem and show that both GD and SGD can achieve global minima of the training loss for any $L\geq 1$, with the aid of over-parameterization and random initialization. At the core of our analysis is to show that Gaussian random initialization followed by (stochastic) gradient descent generates a sequence of iterates within a small perturbation region centering around the initial weights. In addition, we will show that the empirical loss function of deep ReLU networks has very good local curvature properties inside the perturbation region, which guarantees the global convergence of (stochastic) gradient descent. More specifically, our main contributions are summarized as follows:

We show that with Gaussian random initialization on each layer, when the number of hidden nodes per layer is at least \widetilde{\Omega}\big{(}\text{poly}(n,\phi^{-1},L)\big{)}, GD can achieve zero training error within \widetilde{O}\big{(}\text{poly}(n,\phi^{-1},L)\big{)} iterations, where $\phi$ is the data separation distance, $n$ is the number of training examples, and $L$ is the number of hidden layers. Our result can be applied to a broad family of loss functions, as opposed to cross entropy loss studied in Li and Liang (2018) and quadratic loss considered in Du et al. (2018b).

We also prove a similar convergence result for SGD. We show that with Gaussian random initialization on each layer, when the number of hidden nodes per layer is at least \widetilde{\Omega}\big{(}\text{poly}(n,\phi^{-1},L)\big{)}, SGD can also achieve zero training error within \widetilde{O}\big{(}\text{poly}(n,\phi^{-1},L)\big{)} iterations.

In terms of data distribution, we only make the so-called data separation assumption, which is more realistic than the assumption on the gram matrix made in Du et al. (2018b). The data separation assumption in this work is similar, but slightly milder, than that in Li and Yuan (2017) in the sense that it holds as long as the data are sampled from a distribution with a constant margin separating different classes.

When we were preparing this manuscript, we were informed that two concurrent work (Allen-Zhu et al., 2018b; Du et al., 2018a) has appeared on-line very recently. Our work bears a similarity to Allen-Zhu et al. (2018b) in the high-level proof idea, which is to extend the results for two-layer ReLU networks in Li and Liang (2018) to deep ReLU networks. However, while Allen-Zhu et al. (2018b) mainly focuses on the regression problems with least square loss, we study the classification problems for a broad class of loss functions based on a milder data distribution assumption. Du et al. (2018a) also studies the regression problem. Compared to their work, our work is based on a different assumption on the training data and is able to deal with the nonsmooth ReLU activation function.

The remainder of this paper is organized as follows: In Section 2, we discuss the literature that are most related to our work. In Section 3, we introduce the problem setup and preliminaries of our work. In Sections 4 and 5, we present our main theoretical results and their proofs respectively. We conclude our work and discuss some future work in Section 6.

Related Work

Due to the huge amount of literature on deep learning theory, we are not able to include all papers in this big vein here. Instead, we review the following three major lines of research, which are most related to our work.

One-hidden-layer neural networks with ground truth parameters Recently a series of work (Tian, 2017; Brutzkus and Globerson, 2017; Li and Yuan, 2017; Du et al., 2017a, b; Zhang et al., 2018) study a specific class of shallow two-layer (one-hidden-layer) neural networks, whose training data are generated by a ground truth network called “teacher network”. This series of work aim to provide recovery guarantee for gradient-based methods to learn the teacher networks based on either the population or empirical loss functions. More specifically, Tian (2017) proved that for two-layer ReLU networks with only one hidden neuron, GD with arbitrary initialization on the population loss is able to recover the hidden teacher network. Brutzkus and Globerson (2017) proved that GD can learn the true parameters of a two-layer network with a convolution filter. Li and Yuan (2017) proved that SGD can recover the underlying parameters of a two-layer residual network in polynomial time. Moreover, Du et al. (2017a, b) proved that both GD and SGD can recover the teacher network of a two-layer CNN with ReLU activation function. Zhang et al. (2018) showed that GD on the empirical loss function can recover the ground truth parameters of one-hidden-layer ReLU networks at a linear rate.

Deep linear networks Beyond shallow one-hidden-layer neural networks, a series of recent work (Hardt and Ma, 2016; Kawaguchi, 2016; Bartlett et al., 2018; Arora et al., 2018a, b) focus on the optimization landscape of deep linear networks. More specifically, Hardt and Ma (2016) showed that deep linear residual networks have no spurious local minima. Kawaguchi (2016) proved that all local minima are global minima in deep linear networks. Arora et al. (2018b) showed that depth can accelerate the optimization of deep linear networks. Bartlett et al. (2018) proved that with identity initialization and proper regularizer, GD can converge to the least square solution on a residual linear network with quadratic loss function, while Arora et al. (2018a) proved the same properties for general deep linear networks.

Generalization bounds for deep neural networks The phenomenon that deep neural networks generalize better than shallow neural networks have been observed in practice for a long time (Langford and Caruana, 2002). Besides classical VC-dimension based results (Vapnik, 2013; Anthony and Bartlett, 2009), a vast literature have recently studied the connection between the generalization performance of deep neural networks and their architectures (Neyshabur et al., 2015, 2017a, 2017b; Bartlett et al., 2017; Golowich et al., 2017; Arora et al., 2018c; Allen-Zhu et al., 2018a). More specifically, Neyshabur et al. (2015) derived Rademacher complexity for a class of norm-constrained feed-forward neural networks with ReLU activation function. Bartlett et al. (2017) derived margin bounds for deep ReLU networks based on Rademacher complexity and covering number. Neyshabur et al. (2017a, b) also derived similar spectrally-normalized margin bounds for deep neural networks with ReLU activation function using PAC-Bayes approach. Golowich et al. (2017) studied size-independent sample complexity of deep neural networks and showed that the sample complexity can be independent of both depth and width under additional assumptions. Arora et al. (2018c) proved generalization bounds via compression-based framework. Allen-Zhu et al. (2018a) showed that an over-parameterized one-hidden-layer neural network can learn a one-hidden-layer neural network with fewer parameters using SGD up to a small generalization error, while similar results also hold for over-parameterized two-hidden-layer neural networks.

Problem Setup and Preliminaries

2 Problem Setup

where $p\leq 1$ is a positive constant. Note that $a_{0}$ can be $+\infty$.

This assumption holds for a large class of loss functions including hinge loss, cross-entropy loss and exponential loss. It is worthy noting that when $\alpha_{0}=+\infty$ and $p=1/2$, this reduces to Polyak-Łukojasiewicz (PL) condition (Polyak, 1963).

In addition, we make the following assumptions on the training data.

$\|\mathbf{x}_{i}\|_{2}=1$ and $(\mathbf{x}_{i})_{d}=\mu$ for all $i\in\{1,\ldots,n\}$, where $\mu\in(0,1)$ is a constant.

As is shown in the assumption above, the last entry of input $\mathbf{x}$ is considered to be a constant $\mu$, which introduces the bias term in the input layer of the network.

For all $i,i^{\prime}\in\{1,\ldots,n\}$, if $y_{i}\neq y_{i^{\prime}}$, then $\|\mathbf{x}_{i}-\mathbf{x}_{i^{\prime}}\|_{2}\geq\phi$ for some $\phi>0$.

Assumption 3.5 is a weaker version of Assumption 2.1 in Allen-Zhu et al. (2018b), which assumes that every two different data points are separated by a constant. In comparison, Assumption 3.5 only requires that inputs with different labels are separated, which is a much more practical assumption since it holds for all data distributions with margin $\phi$, while the data separation distance in Allen-Zhu et al. (2018b) is usually dependent on the sample size $n$ when the examples are generated independently.

Define $M=\max\{m_{1},\ldots,m_{L}\}$, $m=\min\{m_{1},\ldots,m_{L}\}$. We assume that $M\leq 2m$.

Assumption 3.6 states that the number of nodes at all layers are of the same order. The constant $2$ is not essential and can be replaced with an arbitrary constant greater than or equal to $1$.

3 Optimization Algorithms

In this paper, we consider training a deep neural network with Gaussian initialization followed by gradient descent/stochastic gradient descent.

Gaussian initialization. We say that the weight matrices $\mathbf{W}_{1},\ldots,\mathbf{W}_{L}$ are generated from Gaussian initialization if each column of $\mathbf{W}_{l}$ is generated independently from the Gaussian distribution $N(\mathbf{0},2/m_{l}\mathbf{I})$ for all $l=1,\ldots,L$.

Gradient descent. We consider solving the empirical risk minimization problem (3.4) with gradient descent with Gaussian initialization: let $\mathbf{W}_{1}^{(0)},\ldots,\mathbf{W}_{L}^{(0)}$ be weight matrices generated from Gaussian initialization, we consider the following gradient descent update rule:

where $\eta>0$ is the step size (a.k.a., learning rate).

Stochastic gradient descent. We also consider solving (3.4) using stochastic gradient descent with Gaussian initialization. Again, let $\{\mathbf{W}_{l}^{(0)}\}_{l=1}^{L}$ be generated from Gaussian initialization. At the $k$-th iteration, a minibatch $\mathcal{B}^{(k)}$ of training examples with batch size $B$ is sampled from the training set, and the stochastic gradient is calculated as follows:

The update rule for stochastic gradient descent is then defined as follows:

4 Preliminaries

Here we briefly introduce some useful notations and provide some basic calculations regarding the neural network under our setting.

Output after the $l$-th layer: Given an input $\mathbf{x}_{i}$, the output of the neural network after the $l$-th layer is

Output of the neural network: The output of the neural network with input $\mathbf{x}_{i}$ is as follows:

where we define $\mathbf{x}_{0,i}=\mathbf{x}_{i}$ and the last equality holds for any $l\geq 1$.

Gradient of the neural network: The partial gradient of the training loss $L_{S}(\mathbf{W})$ with respect to $\mathbf{W}_{l}$ is as follows:

Main Theory

In this section, we show that with random Gaussian initialization, over-parameterization helps gradient based algorithms, including gradient descent and stochastic gradient descent, converge to the global minimum, i.e., find some points with arbitrary small training loss.

We provide the following theorem which characterizes the required numbers of hidden nodes and iterations such that the gradient descent can attain the global minimum of the empirical training loss function.

Suppose $\mathbf{W}_{1}^{(0)},\dots,\mathbf{W}_{L}^{(0)}$ are generated by Gaussian initialization. Then under Assumptions 3.1-3.6, if set the step size $\eta=O(n^{-3}L^{-9}m^{-1})$, the number of hidden nodes per layer

then with high probability, gradient descent can find a point $\mathbf{W}^{(K)}$ such that $L_{S}(\mathbf{W}^{(K)})\leq\epsilon$.

Theorem 4.1 suggests that the required number of hidden nodes and the number of iterations are both polynomial in the number of training examples $n$, and the separation parameter $\phi$. This is consistent with the recent work on the global convergence in training neural networks (Li and Yuan, 2017; Du et al., 2018b; Allen-Zhu et al., 2018c; Du et al., 2018a; Allen-Zhu et al., 2018b). Moreover, we prove that the dependence on the number of hidden layers $L$ is also polynomial, which is similar to Allen-Zhu et al. (2018b) and strictly better than Du et al. (2018a), where the dependence on $L$ is proved to be $e^{\Omega(L)}$. Regarding different loss functions (depending on $\alpha_{0}$ and $p$ according to Assumption 3.2), the dependence in $\epsilon$ ranges from $\widetilde{O}(\epsilon^{-1})$ to $\widetilde{O}(1)$.

Based on the results in Theorem 4.1, we are able to characterize the required number of hidden nodes per layer that gradient descent can find a point with zero training error in the following corollary.

2 Stochastic Gradient Descent

where $p\leq 1$ is a positive constant and $\rho_{0}/\alpha_{0}=O(1)$.

Suppose $\mathbf{W}_{1}^{(0)},\dots,\mathbf{W}_{L}^{(0)}$ are generated by Gaussian random. Then under Assumptions 3.1-3.6 and 4.4, if the step size $\eta=O(n^{-3}L^{-9}m^{-1})$, the number of hidden nodes per layer satisfies

then with high probability, stochastic gradient descent can find a point $\mathbf{W}^{(K)}$ such that $L_{S}(\mathbf{W}^{(K)})\leq\epsilon$.

Similar to gradient descent, the following corollary characterizes the required number of hidden nodes per layer that stochastic gradient descent can achieve zero training error.

Theorem 4.5 suggests that, to find the global minimum, both the required number of hidden nodes and the number of iterations for stochastic gradient descent are also polynomial in $n$, $\phi$ and $L$, which matches the result in Allen-Zhu et al. (2018b) for the regression problem. In addition, as it cannot be directly observed in Corollaries 4.3 and 4.6, we remark here that compared with gradient descent, the required numbers of hidden nodes and iterations of stochastic gradient to achieve zero training error is worse by a factor ranging from $O(n^{2})$ to $O(n^{4})$. The detailed comparison can be found in the proofs of Theorems 4.1 and 4.5.

Proof of the Main Theory

In this section, we provide the proof of the main theory, including Theorems 4.1 and 4.5. Our proofs for these two theorems can be decomposed into the following five steps:

We prove the basic properties for Gaussian random matrices $\{\mathbf{W}_{l}\}_{l,\dots,L}$ in Theorem 5.1, which constitutes a basic structure of the neural network after Gaussian random initialization.

Based on Theorem 5.1, we analyze the effect of $\|\cdot\|_{2}$-perturbations on Gaussian initialized weight matrices within a perturbation region with radius $\tau$, and show that the neural network enjoys good local curvature properties in Theorem 5.3.

Based on the assumption that all iterates are within the perturbation region centering at $\mathbf{W}^{(0)}$ with radius $\tau$, we establish the convergence results in Lemmas 5.4 and 5.6, and derive conditions on the product of iteration number $k$ and step size $\eta$ that guarantees convergence.

We show that as long as the product of iteration number $k$ and step size $\eta$ is smaller than some quantity $T$, (stochastic) gradient descent with $k$ iterations remains in the perturbation region centering around the Gaussian initialization $\mathbf{W}^{(0)}$, which justifies the application of Theorem 5.3 to the iterates of (stochastic) gradient descent.

We finalize the proof by ensuring that (stochastic) gradient descent converges before $k\eta$ exceeds $T$ by setting on the number of hidden nodes in each layer $m$ to be large enough.

The following theorem summarizes some high probability results of neural networks with Gaussian random initialization, which is pivotal to establish the subsequent theoretical analyses.

Suppose that $\mathbf{W}_{1},\ldots,\mathbf{W}_{L}$ are generated by Gaussian initialization. Then under Assumptions 3.4, 3.5 and 3.6, there exist absolute constants $\overline{C},\overline{C}^{\prime},\underline{C},\underline{C}^{\prime},\underline{C}^{\prime\prime}>0$ such that for any $\delta>0$, $\beta>0$ and positive integer $s$, as long as

and $\phi\leq\underline{C}\min\{L^{-1},\mu\}$, with probability at least $1-\delta$, all the following results hold:

\big{|}\|\mathbf{x}_{l,i}\|_{2}-1\big{|}\leq\overline{C}^{\prime}L\sqrt{\log(nL/\delta)/m},\|\mathbf{W}_{l}\|_{2}\leq\overline{C}^{\prime} for all $l=1\ldots,L$ and $i=1,\ldots,n$.

\big{\|}\|\mathbf{x}_{l,i}\|_{2}^{-1}\mathbf{x}_{l,i}-\|\mathbf{x}_{l,i^{\prime}}\|_{2}^{-1}\mathbf{x}_{l,i^{\prime}}\big{\|}_{2}\geq\phi/2 for all $l=1,\ldots,L$ and $i,i^{\prime}\in\{1,\ldots,n\}$ such that $y_{i}\neq y_{i^{\prime}}$.

$|\widehat{y}_{i}|\leq\overline{C}^{\prime}\sqrt{\log(n/\delta)}$ for all $i=1,\ldots,n$.

\big{|}\{j\in[m_{l}]:|\langle\mathbf{w}_{l,j},\mathbf{x}_{l-1,i}\rangle|\leq\beta\}\big{|}\leq 2m_{l}^{3/2}\beta for all $l=1,\ldots,L$ and $i=1,\ldots,n$.

\big{\|}\mathbf{W}_{l_{2}}^{\top}\big{(}\prod_{r=l_{1}}^{l_{2}-1}\bm{\Sigma}_{r,i}\mathbf{W}_{r}^{\top}\big{)}\big{\|}_{2}\leq\overline{C}^{\prime}L for all $1\leq l_{1}<l_{2}\leq L$ and $i=1,\ldots,n$.

\mathbf{v}^{\top}\big{(}\prod_{r=l}^{L}\bm{\Sigma}_{r,i}\mathbf{W}_{r}^{\top}\big{)}\mathbf{a}\leq\overline{C}^{\prime}L\sqrt{s\log(M)} for all $l=1,\ldots,L$, $i=1,\ldots,n$ and all $\mathbf{a}\in S^{m_{l-1}-1}$ with $\|\mathbf{a}\|_{0}\leq s$ .

Theorem 5.1 summarizes all the properties we need for Gaussian initialization. In the sequel, we always assume that results (i)-(viii) hold for the Gaussian initialization. The parameters $\beta$ and $s$ in Theorem 5.1 are introduced to characterize the activation pattern of the ReLU activation functions in each layer. Their values that directly help the final convergence proof is derived during the proof of Theorem 5.3 as $\beta=O(L^{4/3}\tau^{2/3}m^{-1/2})$ and $s=O(L^{4/3}\tau^{2/3}m)$, where $\tau=\widetilde{O}(n^{-9}L^{-17}\phi^{3})$ is the perturbation level. Therefore, the condition on the rate of $m_{1},\ldots,m_{L}$ given by (5.1) is satisfied under the final assumptions on $m$ given in Theorem 4.1 and Theorem 4.5.

We perform $\|\cdot\|_{2}$-perturbation on the collection of random matrices $\{\mathbf{W}_{l}\}_{l=1,\dots,L}$ with perturbation level $\tau$, which formulates a perturbation region centering at $\{\mathbf{W}_{l}\}_{l=1,\dots,L}$ with radius $\tau$. Let $\widetilde{\mathbf{W}}=\{\widetilde{\mathbf{W}}_{1},\ldots,\widetilde{\mathbf{W}}_{L}\}$ and $\widehat{\mathbf{W}}=\{\widehat{\mathbf{W}}_{1},\ldots,\widehat{\mathbf{W}}_{L}\}$ be two collections of weight matrices. For $l=1,\ldots,L$, denote $\widetilde{\mathbf{x}}_{l,i}$, $\widehat{\mathbf{x}}_{l,i}$ be the output of the $l$-th hidden layer of the ReLU network with input $\mathbf{x}_{i}$ and weight matrices $\widetilde{\mathbf{W}}$ and $\widehat{\mathbf{W}}$ respectively. Define $\widetilde{\mathbf{x}}_{0,i}=\widetilde{\mathbf{x}}_{0,i}=\mathbf{x}_{i}$, and

for all $l=1,\ldots,L$. We summarize their properties in the following theorem.

Suppose that $\mathbf{W}_{1},\ldots,\mathbf{W}_{L}$ are generated via Gaussian initialization, and all results (i)-(viii) in Theorem 5.1 hold. Let $\{\widetilde{\mathbf{W}}_{l}\}_{l=1,\dots,L}$, $\{\widehat{\mathbf{W}}_{l}\}_{l=1,\dots,L}$ be perturbed weight matrices satisfying $\|\widetilde{\mathbf{W}}_{l}-\mathbf{W}_{l}\|_{2},\|\widehat{\mathbf{W}}_{l}-\mathbf{W}_{l}\|_{2}\leq\tau$, $l=1,\ldots,L$. Then under Assumptions 3.5 and 3.6, there exist absolute constants $\overline{C},\overline{C}^{\prime},\underline{C},\underline{C}^{\prime}$, such that as long as $\tau\leq\underline{C}\{[L^{-11}(\log(M))^{-3/2}]\land(\phi^{3/2}n^{-3}L^{-2})\}$, the following results hold:

$\|\widetilde{\mathbf{W}}_{l}\|_{2}\leq\overline{C}$ for all $l\in[L]$.

$\|\widehat{\mathbf{x}}_{l,i}-\widetilde{\mathbf{x}}_{l,i}\|_{2}\leq\overline{C}L\cdot\sum_{r=1}^{l}\|\widehat{\mathbf{W}}_{r}-\widetilde{\mathbf{W}}_{r}\|_{2}$ for all $l\in[L]$ and $i\in[n]$.

$\|\widehat{\bm{\Sigma}}_{l,i}-\widetilde{\bm{\Sigma}}_{l,i}\|_{0}\leq\overline{C}L^{4/3}\tau^{2/3}m_{l}$ for all $l\in[L]$ and $i\in[n]$.

\big{|}\{j\in[m_{L}]:\text{there exists }i\in[n]\mbox{ such that }(\widetilde{\bm{\Sigma}}_{L,i}-\bm{\Sigma}_{L,i})_{jj}\neq 0\}\big{|}\leq\overline{C}nL^{4/3}\tau^{2/3}m_{L}.

\big{\|}\prod_{r=l_{1}}^{l_{2}}\widetilde{\bm{\Sigma}}_{r,i}\widetilde{\mathbf{W}}_{r}^{\top}\big{\|}_{2}\leq\overline{C}L for all $1\leq l_{1}<l_{2}\leq L$.

The squared Frobenius norm of the partial gradient with respect to the weight matrix in the last hidden layer has the following lower bound:

where $\widetilde{y}_{i}=f_{\widetilde{\mathbf{W}}}(\mathbf{x}_{i})$.

The spectral norms of gradients and stochastic gradients at each layer have the following upper bounds:

where $\widetilde{y}_{i}=f_{\widetilde{\mathbf{W}}}(\mathbf{x}_{i})$, $B=|\mathcal{B}|$ denotes the minibatch size in SGD.

The gradient lower bound provided in (vii) implies that within the perturbation region, the empirical loss function of deep neural network enjoys good local curvature properties, which play an essential role in the convergence proof of (stochastic) gradient descent. The gradient upper bound in (viii) quantifies how much the weight matrices of the neural network would change during (stochastic) gradient descent, which is utilized to guarantee that the weight matrices won’t escape from the perturbation region during the training process.

We organize our proof as the following three steps: (1) we first assume that during gradient descent, each iterate is in the preset perturbation region centering at $\mathbf{W}^{(0)}$ with radius $\tau$, and use the results in Theorem 5.3 to establish the convergence guarantee; (2) we prove the upper bound of the number of iteration such that the distance between the iterate and the initial point does not exceed $\tau$; (3) we compute the minimum number of hidden nodes such that gradient descent achieves the target accuracy before exceeding the upper bound derived in step (2).

For step (1), the following lemma provides the convergence guarantee of gradient descent while assuming all iterates are in the preset perturbation region, i.e., $\|\mathbf{W}_{l}^{(k)}-\mathbf{W}_{l}^{(0)}\|_{2}\leq\tau$ for all $l=1,\ldots,L$ and $k=1,\ldots,K$.

Suppose that $\mathbf{W}_{1}^{(0)},\dots,\mathbf{W}_{L}^{(0)}$ are generated via Gaussian initialization, and all results (i)-(viii) in Theorem 5.1 hold. Under Assumptions 3.1-3.6, if $\|\mathbf{W}_{l}^{(k)}-\mathbf{W}_{l}^{(0)}\|_{2}\leq\tau$ for all $l=1,\ldots,L$ and $k=1,\ldots,K$ with perturbation level $\tau=O(n^{-9}L^{-17}\phi^{3})$, the step size $\eta=O(n^{-3}L^{-9}m^{-1}\phi)$ and

when $\alpha_{0}<\infty$, then gradient descent is able to find a point $\mathbf{W}^{(K)}$ such that $L_{S}(\mathbf{W}^{(K)})\leq\epsilon$.

The following lemma provides the upper bound of the iteration number such that the distance between the iterate and the initial point does not exceed the perturbation radius $\tau$.

Suppose that $\mathbf{W}_{1}^{(0)},\dots,\mathbf{W}_{L}^{(0)}$ are generated by Gaussian initialization, and all results (i)-(viii) in Theorem 5.1 hold. Then there exist a constant $T=O(L^{-4}n^{-3}\tau^{2}\phi)$ such for all iteration number $k$ and step size $\eta$ satisfying $k\eta\leq T$, it holds that $\|\mathbf{W}_{l}^{(k)}-\mathbf{W}_{l}^{(0)}\|_{2}\leq\tau$ for all $l\in[L]$.

Now we are ready to prove Theorem 4.1 based on Lemmas 5.4 and 5.5.

The proof is straightforward. By Lemma 5.5, it suffices to show that the lower bound of $K\eta$ derived in Lemma 5.4 is smaller than $T=O(L^{-4}n^{-3}\tau^{2}\phi)=O(L^{-38}n^{-21}\phi^{7})$, where we plug in the assumption that $\tau=O(n^{-9}L^{-17}\phi^{3})$. Therefore, we can derive the following lower bound on the number of hidden nodes per layer, i.e., $m$:

Moreover, the required number of iterations, i.e., $K$ can be directly derived by combining the results of $K\eta$ in Lemma 5.4 and the choice of the step size $\eta=O(n^{-3}L^{-9}m^{-1}\phi)$, thus we omit the detail here. ∎

2 Proof of Theorem 4.5

Similar to the proof for gradient descent, we first deliver the following lemma which characterizes the convergence of stochastic gradient descent for the training of ReLU network under the assumption that all iterates are in the preset perturbation region.

Suppose that $\mathbf{W}_{1}^{(0)},\dots,\mathbf{W}_{L}^{(0)}$ are generated by Gaussian initialization, and all results (i)-(viii) in Theorem 5.1 hold. Under Assumptions 3.1-3.6 and 4.4, if $\|\mathbf{W}_{l}^{(k)}-\mathbf{W}_{l}^{(0)}\|_{2}\leq\tau$ for all $l=1,\ldots,L$ and $k=1,\ldots,K$ with perturbation level $\tau=O(n^{-9}L^{-17}\phi^{3})$, the step size $\eta=O(Bn^{-4}L^{-9}m^{-1}\phi)$, and

when $\alpha_{0}<\infty$, then stochastic gradient descent is able to find a point $\mathbf{W}^{(K)}$ such that $L_{S}(\mathbf{W}^{(K)})\leq\epsilon$.

The following lemma provides the upper bound of the iteration number such that the distance between the iterate and the initial point does not exceed the perturbation radius $\tau$.

Suppose that $\mathbf{W}_{1}^{(0)},\dots,\mathbf{W}_{L}^{(0)}$ are generated by Gaussian initialization, and all results (i)-(viii) in Theorem 5.1 hold. If the step size $\eta=O(\phi m^{-1}n^{2p-7}L^{-8}B^{2})$, then there exists $T$ with rate $T=\widetilde{O}(L^{-2}n^{-1}Bm^{-1/2}\tau)$ when $\alpha_{0}=\infty$ and $T=O(L^{-2}m^{-1/2}\tau)$ when $\alpha<\infty$ such that for all iteration number $k$ satisfying $k\eta\leq T$, with high probability it holds that $\|\mathbf{W}_{l}^{(k)}-\mathbf{W}_{l}^{(0)}\|_{2}\leq\tau$ for all $l\in[L]$.

Similar to the proof of Theorem 4.1, the minimum required number of hidden nodes per layer can be derived by setting the lower bound of $K\eta$ in Lemma 5.6 to be smaller than $T=\widetilde{O}(L^{-2}n^{-1}m^{-1/2}\tau)=\widetilde{O}(n^{-10}L^{-19}m^{-1/2}\phi^{3})$ when $\alpha_{0}=\infty$ or $T=O(L^{-2}m^{-1/2}\tau)=O(n^{-9}L^{-19}m^{-1/2}\phi^{3})$ when $\alpha_{0}<\infty$. Therefore the minimum required number of hidden nodes per layer satisfies

We now proceed to derive the required iteration numbers. By Lemmas 5.6 and 5.7, we can set the step size to be $\eta=O(\phi m^{-1}n^{2p-7}L^{-9}B^{2})$. Then using the bound of $K\eta$ derived in Lemma 5.6, we have

Conclusions and Future Work

In this paper, we studied training deep neural networks by gradient descent and stochastic gradient descent. We proved that both gradient descent and stochastic gradient descent can achieve global minima of over-parameterized deep ReLU networks with random initialization, for a general class of loss functions, with only mild assumption on training data. Our theory sheds light on understanding why stochastic gradient descent can train deep neural networks very well in practice, and paves the way to study the optimization dynamics of training more sophisticated deep neural networks.

In the future, we will sharpen the polynomial dependence of our results on those problem-specific parameters.

Acknowledgment

We would like to thank Spencer Frei for helpful comments on the first version of this paper.

Appendix A Proof of Theorem 5.1

In this section we provide the proof of Theorem 5.1. The bound for $\|\mathbf{W}_{l}\|_{2}$ given by result (i) in Theorem 5.1 follows from standard results for Gaussian random matrices with independent entries (See Corollary 5.35 in Vershynin (2010)). We split the rest results into several lemmas and prove them separately.

We first give the bound for the norms of the outputs of each layer. Intuitively, since the columns of $\mathbf{W}_{l}$ are sampled independently from $N(\mathbf{0},2/m_{l}\mathbf{I})$, given the output of the previous layer $\mathbf{x}_{l-1,i}$, the expectation of $\|\mathbf{W}_{l}^{\top}\mathbf{x}_{l-1,i}\|_{2}^{2}$ is $2\|\mathbf{x}_{l-1,i}\|_{2}^{2}$. Moreover, the ReLU activation function truncates roughly half of the entries of $\mathbf{W}_{l}^{\top}\mathbf{x}_{l-1,i}$ to zero, and therefore $\|\mathbf{x}_{l,i}\|_{2}^{2}$ should be approximately equal to $\|\mathbf{x}_{l-1,i}\|_{2}^{2}$. This leads to Lemma A.1 and Corollary A.2.

Denote by $\mathbf{w}_{l,j}$ the $j$-th column of $\mathbf{W}_{l}$. Suppose that for any $l=1,\ldots,L$, $\mathbf{w}_{l,1},\ldots,\mathbf{w}_{l,m_{l}}$ are generated independently from $N(\mathbf{0},2/m_{l}\mathbf{I})$. Then there exists an absolute constant $C$ such that for any $\delta>0$, as long as $m_{l}\geq C^{2}\log(nL/\delta)$, with probability at least $1-\delta$,

for all $i=1,\ldots,n$ and $l=1,\ldots,L$.

For any fixed $i\in\{1,\ldots,n\}$, $l\in\{1,\ldots,L\}$ and $j\in\{1,\ldots,m_{l}\}$, condition on $\mathbf{x}_{l-1,i}$ we have $\mathbf{w}_{l,j}^{\top}\mathbf{x}_{l-1,i}\sim N(\mathbf{0},2\|\mathbf{x}_{l-1,i}\|_{2}^{2}/m_{l})$. Therefore,

Since $\|\mathbf{x}_{l,i}\|_{2}^{2}=\sum_{j=1}^{m_{l}}\sigma^{2}(\mathbf{w}_{l,j}^{\top}\mathbf{x}_{l-1,i})$ and condition on $\mathbf{x}_{l-1}$, $\|\sigma^{2}(\mathbf{w}_{l,j}^{\top}\mathbf{x}_{l-1,i})\|_{\psi_{1}}\leq C_{1}\|\mathbf{x}_{l-1,i}\|_{2}^{2}/m_{l}$ for some absolute constant $C_{1}$, by Bernstein inequality (See Proposition 5.16 in Vershynin (2010)), for any $\xi\geq 0$ we have

Taking union bound over $l$ and $i$ gives

The inequality above further implies that if $m_{l}\geq C_{3}^{2}\log(nL/\delta)$, then with probability at least $1-\delta$, we have

for any $i=1,\ldots,n$ and $l=1,\ldots,L$, where $C_{3}$ is an absolute constant. This completes the proof. ∎

Under the same conditions as Lemma A.1, with probability at least $1-\delta$,

where $m=\min\{m_{1},\ldots,m_{L}\}$, and $C$ is an absolute constant.

The result directly follows by Lemma A.1 and induction. ∎

By Corollary A.2, we can see that the norms of the inputs are roughly preserved after passing through layers in the ReLU neural network with properly scaled Gaussian random weights. We now proceed to show (ii) in Theorem 5.1, which states that the inner product of any two samples, although may not be preserved throughout layers, also share a common upper bound based on the Assumption 3.5. The detailed results are given in Lemmas A.3, A.4 and Corollary A.5 below.

For $I_{1}$, it follows by direct calculation that $I_{1}=(1-\alpha^{2})^{3/2}/(2\pi)=O(\theta^{3})$. For $I_{2}$, integration by parts gives

For $\arccos$ function we have the following expansion:

Therefore when $\alpha=1-\theta^{2}/2$ for small enough constant $\theta$, we have

Combining the obtained rates with (A.1) completes the proof. ∎

Suppose that $\mathbf{w}_{l,1},\ldots,\mathbf{w}_{l,m_{l}}$ are generated independently from $N(\mathbf{0},2/m_{l}\mathbf{I})$. Let $\overline{\mathbf{x}}_{l-1,i}=\mathbf{x}_{l-1,i}/\|\mathbf{x}_{l-1,i}\|_{2}$, $i=1,\ldots,n$. For any fixed $i,i^{\prime}\in\{1,\ldots,n\}$, if $\phi\leq\kappa L^{-1}$ for some small enough absolute constant $\kappa$, then for any $\delta>0$, if $m=\min\{m_{1},\ldots,m_{L}\}\geq CL^{4}\phi^{-4}\log(4n^{2}L/\delta)$ for some large enough absolute constant $C$, with probability at least $1-\delta$,

for all $i,j=1,\ldots,n$, $l=1,\ldots,L$.

We first consider any fixed $l\geq 1$. Suppose that $\|\overline{\mathbf{x}}_{l-1,i}-\overline{\mathbf{x}}_{l-1,i^{\prime}}\|_{2}\geq[1-(2L)^{-1}\log(2)]^{l-1}\phi$. If we can show that under this condition, with high probability

then the result of the lemma follows by union bound and induction. Denote

Then by assumption we have $\|\overline{\mathbf{x}}_{l-1,i}-\overline{\mathbf{x}}_{l-1,i^{\prime}}\|_{2}^{2}\geq\phi_{l-1}^{2}$. Therefore $\overline{\mathbf{x}}_{l-1,i}^{\top}\overline{\mathbf{x}}_{l-1,i^{\prime}}\leq 1-\phi_{l-1}^{2}/2$. It follows by direct calculation that

By Lemma A.3 and the assumption that $\phi_{l-1}\leq\phi\leq\kappa$, we have

Condition on $\mathbf{x}_{l-1,i}$ and $\mathbf{x}_{l-1,i^{\prime}}$, by Lemma 5.14 in Vershynin (2010) we have

where $C_{1}$ is an absolute constant. Therefore if $m_{l}\geq C_{2}^{2}\log(4n^{2}L/\delta)$, similar to the proof of Lemma A.1, by Bernstein inequality and union bound, with probability at least $1-\delta/(4n^{2}L)$ we have

where $C_{2}$ is an absolute constant. Therefore with probability at least $1-\delta/(4n^{2}L)$ we have

By union bound and Lemma A.1, if $m_{r}\geq C_{3}L^{4}\phi_{l}^{-4}\log(4n^{2}L/\delta)$, $r=1,\ldots,l$ for some large enough absolute constant $C_{3}$ and $\phi\leq\kappa L^{-1}$ for some small enough absolute constant $\kappa$, then with probability at least $1-\delta/(2n^{2}L)$ we have

Moreover, by Lemma A.1, with probability at least $1-\delta/(2n^{2}L)$ we have

and therefore with probability at least $1-\delta/(n^{2}L)$, we have

Applying union bound and induction over $l=1,\ldots,L$ completes the proof. ∎

Under the same conditions in Lemma A.4, with probability at least $1-\delta$,

for all $i,i^{\prime}=1,\ldots,n$, $l=1,\ldots,L$.

It directly follows by Lemma A.4 and the fact that $1-x/2\geq e^{-x}$ for $x\in[0,\log(2)]$. ∎

The following lemma gives the bound of $\widehat{y}_{i}$. It relies on the fact that half of the entries of $\mathbf{v}$ are $1$’s and the other half are $-1$’s.

Suppose that $\{\mathbf{W}_{l}\}_{l=1}^{L}$ are generated via Gaussian initialization. Then for any $\delta>0$, with probability at least $1-\delta$, it holds that

for all $i=1,\ldots,n$, where $C$ is an absolute constant.

Apparently, we have $\|\sigma(\mathbf{w}_{L,j}^{\top}\mathbf{x}_{L-1,i})-\sigma(\mathbf{w}_{L,j+m_{L}/2}^{\top}\mathbf{x}_{L-1,i})\|_{\psi_{2}}\leq C_{1}m_{L}^{-1/2}$ for some absolute constant $C_{1}$. Therefore by Hoeffding’s inequality and Corollary A.2, with probability at least $1-\delta$, it holds that

We now prove (iv) in Theorem 5.1, which characterizes the activation pattern of ReLU networks with a parameter $\beta$. We summarize the result in Lemma A.7.

For any $\delta>0$, if $m_{l}\geq C\max\{[\beta^{-1}\log(nL/\delta)]^{2/3},L^{2}\log(nL/\delta)\}$ for some large enough constant $C>0$, then with probability at least $1-\delta$, $|{\mathcal{S}}_{l,i}(\beta)|\leq 2m_{l}^{3/2}\beta$ for all $l=1,\ldots,L$ and $i=1,\ldots,n$.

For fixed $l\in\{1,\ldots,L\}$ and $i\in\{1,\ldots,n\}$, define $Z_{j}=\operatorname{\mathds{1}}\{|\langle\mathbf{w}_{l,j},\mathbf{x}_{l-1,i}\rangle|\leq\beta\}$. Note that by Corollary A.2, with probability at least $1-\exp[-\Omega(m/L^{2})]$ we have $\|\mathbf{x}_{l-1,i}\|_{2}\geq 1/2$. Condition on $\mathbf{x}_{l-1,i}$, we have

Then by Bernstein inequality, with probability at least $1-\exp(-\Omega(m_{l}^{3/2}\beta))$,

Applying union bound over $l=1,\ldots,L$, $i=1,\ldots,n$ and using the assumption that $m_{l}^{3/2}\beta\geq C\log(nL/\delta)$ completes the proof. ∎

We now prove (v)-(vii) in Theorem 5.1. These results together characterize the Lipschitz continuity of the gradients. To show these results, we utilize similar proof technique we used in Lemma A.1 and combine the resulting bound with covering number arguments. The details are given in Lemmas A.8, A.9 and A.10.

For any $i=1,\ldots,n$ and $1\leq l_{1}<l_{2}\leq L$, define

If $m\geq C\log(nL^{2}/\delta)$ for some absolute constant $C$, then with probability at least $1-\delta$ we have

for all $i=1,\ldots,n$ and $1\leq l_{1}<l_{2}\leq L$, where $C^{\prime}$ is an absolute constant.

Denote $l_{0}=l_{1}-1$, $\delta^{\prime}=\delta/(nL^{2})$ and

Let $\mathcal{N}_{1}=\mathcal{N}[S^{m_{l_{0}}-1},1/4]$, $\mathcal{N}_{2}=\mathcal{N}[S^{m_{l_{2}}-1},1/4]$ be $1/4$-nets covering $S^{m_{l_{0}}-1}$ and $S^{m_{l_{2}}-1}$ respectively. Then by Lemma 5.2 in Vershynin (2010), we have

where $\bm{\alpha}_{\mathbin{\!/\mkern-5.0mu/\!}}=\bm{\alpha}^{\top}\mathbf{x}_{l-1,i}/\|\mathbf{x}_{l-1,i}\|_{2}^{2}\cdot\mathbf{x}_{l-1,i}$ and $\bm{\alpha}_{\perp}=\bm{\alpha}-\bm{\alpha}_{\!/\mkern-5.0mu/\!}$. Therefore, similar to the proof of Lemma A.1 and Corollary A.2, with probability at least $1-\delta^{\prime}/2$ we have

for all $\widehat{\mathbf{a}}\in\mathcal{N}_{1}$, where $C_{0}$ is an absolute constant. Therefore by Assumption 3.6 and the assumption that $m\geq C\log(1/\delta^{\prime})$ for some absolute constant $C$, with probability at least $1-\delta^{\prime}$, we have

where $C_{4}$ is an absolute constant. Applying union bound over all $i=1,\ldots,n$ and all $1\leq l_{1}<l_{2}\leq L$ completes the proof. ∎

For any $i=1,\ldots,n$ and $1\leq l\leq L$, If $s\geq C\log(nL/\delta)$ for some absolute constant $C$, then with probability at least $1-\delta$, we have

for all $i=1,\ldots,n$ and $l=1,\ldots,L$, where $C^{\prime}$ is an absolute constant.

then similar to the proof of Lemma A.8, with probability at least $1-\delta^{\prime}$,

for all $\widehat{\mathbf{a}}\in\mathcal{N}$, where $C_{1},C_{2}$ are absolute constants. For any $\mathbf{a}\in S^{m_{l_{0}}-1}$ with at most $s$ non-zero entries, there exists $\widehat{\mathbf{a}}\in\mathcal{N}$ such that $\|\mathbf{a}-\widehat{\mathbf{a}}\|_{2}\leq 1/2$. Therefore, we have

Since $g(\mathbf{a})$ is linear and $\|\mathbf{v}\|_{2}=\sqrt{m_{L}}$, we have

where $C_{3}$ is an absolute constant. Applying union bound over all $i=1,\ldots,n$ and all $1\leq l\leq L$ completes the proof. ∎

For any $i=1,\ldots,n$ and $1\leq l_{1}<l_{2}\leq L$, define

If $s\geq C\log(nL^{2}/\delta)$ for some absolute constant $C$, then with probability at least $1-\delta$ we have

for all $i=1,\ldots,n$ and $1\leq l_{1}<l_{2}\leq L$, where $C^{\prime}$ is an absolute constant.

Similar to previous proofs, it can be shown that with probability at least $1-\delta^{\prime}/2$,

for all $\widehat{a}\in\mathcal{N}_{1}$, where $C_{0}$ is an absolute constant. Therefore with probability at least $1-\delta^{\prime}$, we have

where $C_{4}$ is an absolute constant. Applying union bound over all $i=1,\ldots,n$ and all $1\leq l_{1}<l_{2}\leq L$ completes the proof. ∎

We now prove (viii) in Theorem 5.1. In order to prove this result, we first show that the inner products between normalized hidden layer outputs are lower bounded by a constant related to $\mu$.

The proof follows by direct calculation. By definition, we have

Suppose that $\mathbf{W}_{1},\ldots,\mathbf{W}_{L}$ are generated via Gaussian initialization. Let $\overline{\mathbf{x}}_{l,i}=\mathbf{x}_{l,i}/\|\mathbf{x}_{l,i}\|_{2}$, $i=1,\ldots,n$. For any $\delta>0$, if $m=\geq CL^{4}\mu^{-4}\log(n^{2}L/\delta)$ for some large enough absolute constant $C$, then with probability at least $1-\delta$,

for all $i,i^{\prime}\in\{1,\ldots,n\}$ and all $l\in[L]$.

We prove the result by induction. Suppose that $\overline{\mathbf{x}}_{l,i}^{\top}\overline{\mathbf{x}}_{l,i}\geq\mu^{2}(1-(2L)^{-1}\log(2))^{l}$. Then by Lemma A.11, we have

Condition on $\mathbf{x}_{l-1,i}$ and $\mathbf{x}_{l-1,i^{\prime}}$, we have

where $C_{1},C_{2}$ are absolute constants. Note that we have $[1-(2L)^{-1}\log(2)]^{l}\geq[1-(2L)^{-1}\log(2)]^{L}\geq 1/2$. Therefore if $m\geq CL^{4}\mu^{-4}\log(n^{2}L/\delta)$ for some large enough constant $C$, then by Bernstein inequality, union bound and (i), with probability at least $1-\delta$ we have

Suppose that $\mathbf{W}_{1},\ldots,\mathbf{W}_{L}$ are generated via Gaussian initialization. Let $\overline{\mathbf{x}}_{l,i}=\mathbf{x}_{l,i}/\|\mathbf{x}_{l,i}\|_{2}$, $i=1,\ldots,n$. For any $\delta>0$, if $m\geq CL^{4}\mu^{-4}\log(n^{2}L/\delta)$ for some large enough absolute constant $C$, then with probability at least $1-\delta$,

for all $i,i^{\prime}\in\{1,\ldots,n\}$ and all $l\in[L]$.

It directly follows by Lemma A.12 and the fact that $1-x/2\geq e^{-x}$ for $x\in[0,\log(2)]$. ∎

where $\mathbf{u}^{\prime}:=(u_{2},\ldots,u_{d})^{\top}$ is independent of $u_{1}$. We define the following two events based on a parameter $\gamma\in(0,1]$:

Moreover, by definition, for any $i=1,\ldots,n$ we have

Let $\mathcal{I}=\{i:\|\mathbf{z}_{i}-\mathbf{z}_{1}\|_{2}\geq\widetilde{\phi}\}$. By the assumption that $\phi\leq\widetilde{\mu}/2$, for any $i\in\mathcal{I}$, we have

and if $\widetilde{\phi}^{2}\leq 2$, then

By union bound over $\mathcal{I}$, we have

where the last equality follows from the fact that conditioning on event $\mathcal{E}$, for all $i\in\mathcal{I}$, it holds that $|\langle\mathbf{Q}^{\prime}\mathbf{u}^{\prime},\mathbf{z}_{i}\rangle|\geq|u_{1}|\geq|u_{1}\langle\mathbf{z}_{1},\mathbf{z}_{i}\rangle|$. We then consider two cases: $u_{1}>0$ and $u_{1}<0$, which occur equally likely conditioning on $\mathcal{E}$. Therefore we have

For any $u_{1}^{(1)}>0$ and $u_{1}^{(2)}<0$, denote $\mathbf{w}_{1}=u_{1}^{(1)}\mathbf{z}_{1}+\mathbf{Q}^{\prime}\mathbf{u}^{\prime}$, $\mathbf{w}_{2}=u_{1}^{(2)}\mathbf{z}_{1}+\mathbf{Q}^{\prime}\mathbf{u}^{\prime}$. We now proceed to give lower bound for $\|\mathbf{h}(\mathbf{w}_{1})-\mathbf{h}(\mathbf{w}_{2})\|_{2}$. By (A), we have

Note that for all $i\in\mathcal{I}^{\prime}$, we have $y_{i}=y_{1}$ and $\langle\mathbf{z}_{1},\mathbf{z}_{i}\rangle\geq 1-\widetilde{\phi}^{2}/2\geq 0$. Therefore, since $u_{1}^{(1)}>0>u_{1}^{(2)}$, we have

Therefore $a_{i}^{\prime}\geq 0$ for all $i\in\mathcal{I}^{\prime}$ and

We have shown that $\langle\mathbf{z}_{i},\mathbf{z}_{1}\rangle\geq 0$ for all $i\in\mathcal{I}^{\prime}$. Therefore we have

Since the inequality above holds for any $u_{1}^{(1)}>0$ and $u_{1}^{(2)}<0$, taking infimum gives

where $C$ and $C^{\prime}$ are absolute constants. This completes the proof. ∎

For any given $j\in\{1,\ldots,m_{L}\}$ and $\widehat{\mathbf{a}}$ with $\|\widehat{\mathbf{a}}\|_{\infty}=1$, by Lemma A.14, we have

Let $p_{\phi}=C_{2}\phi/n$. Then by Bernstein inequality and union bound, with probability at least $1-\exp[-C_{3}m_{L}p_{\phi}+n\log(4n/C_{1})]\geq 1-\exp(C_{4}m_{L}\phi/n)$, we have

where $C_{3},C_{4}$ are absolute constants. For any $\mathbf{a}\in S_{\infty,+}^{n-1}$, there exists $\widehat{\mathbf{a}}\in\mathcal{N}$ such that

By (A.7) and (A.8), it is clear that with probability at least $1-\exp(C_{4}m_{L}\phi/n)$, for any $\mathbf{a}\in S_{\infty,+}^{n-1}$, there exist at least $m_{L}p_{\phi}/2$ nodes on layer $L$ that satisfy

Appendix B Proof of Theorem 5.3

It is clear that the bound on $\widetilde{\mathbf{W}}_{l}$ can be proved trivially, since by triangle inequality we have $\|\widetilde{\mathbf{W}}\|_{2}\leq\|\mathbf{W}\|_{2}+\|\widetilde{\mathbf{W}}-\mathbf{W}\|_{2}\leq C$ for some large enough absolute constant $C$. In the following we directly use this result without referring to Theorem 5.3. Similar to the proof of Theorem 5.1, we split the rest results into the following lemmas and prove them separately.

Suppose that $\mathbf{W}_{1},\ldots,\mathbf{W}_{L}$ are generated via Gaussian initialization, and all results (i)-(viii) in Theorem 5.1 hold. For $\tau>0$, let $\widetilde{\mathbf{W}}_{1},\ldots,\widetilde{\mathbf{W}}_{L}$ with $\|\widetilde{\mathbf{W}}_{l}-\mathbf{W}_{l}\|_{2}\leq\tau$, $l=1,\ldots,L$ be the perturbed matrices. Let $\widetilde{\bm{\Sigma}}_{l,i}$, $l=1,\ldots,L$, $i=1,\ldots,n$ be diagonal matrices satisfying $\|\widetilde{\bm{\Sigma}}_{l,i}-\bm{\Sigma}_{l,i}\|_{0}\leq s$ and $|(\widetilde{\bm{\Sigma}}_{l,i}-\bm{\Sigma}_{l,i})_{jj}|,|(\widetilde{\bm{\Sigma}}_{l,i})_{jj}|\leq 1$ for all $l=1,\ldots,L$, $i=1,\ldots,n$ and $j=1,\ldots,m_{l}$. If $\tau,\sqrt{s\log(M)/m}\leq\kappa L^{-3}$ for some small enough absolute constant $\kappa$, then

for any $1\leq l_{1}<l_{2}\leq L$, where $C$ is an absolute constant.

Therefore, let $\mathcal{A}_{r,i}=\{\bm{\Sigma}_{r,i}\mathbf{W}_{r}^{\top},(\widetilde{\bm{\Sigma}}_{r,i}-\bm{\Sigma}_{r,i})\mathbf{W}_{r}^{\top},\widetilde{\bm{\Sigma}}_{r,i}(\widetilde{\mathbf{W}}_{r}-\mathbf{W}_{r})^{\top}\}$, $r=l_{1},\ldots,l_{2}$, then we have

For the rest of the proof, we denote by $|\bm{\Sigma}|$ the diagonal matrix with absolute values of elements of $\bm{\Sigma}$ on the corresponding entries. For each sequence $\mathbf{A}_{l_{1},i},\ldots,\mathbf{A}_{l_{2},i}$, denote

When $\mathbf{A}_{r,i}=\bm{\Sigma}_{r,i}\mathbf{W}_{r}$ for all $r=l_{1},\ldots,l_{2}$, then the bound of $\|\prod_{r=l_{1}}^{l_{2}}\mathbf{A}_{r,i}\|_{2}$ is given by (v) in Theorem 5.1. For all the other terms in the expansion, we consider sequences of the form $\mathbf{B}_{r_{2},i}(\prod_{r=r_{1}+1}^{r_{2}-1}\bm{\Sigma}_{r,i}\mathbf{W}_{r}^{\top})\mathbf{B}_{r_{1},i}$, where

By (vii) in Theorem 5.1, there exists an absolute constant $C_{1}$ such that $\|\mathbf{B}_{r_{2},i}(\prod_{r=r_{1}+1}^{r_{2}-1}\bm{\Sigma}_{r,i}\mathbf{W}_{r}^{\top})\mathbf{B}_{r_{1},i}\|_{2}$ with different choices of $\mathbf{B}_{r_{1},i}$ and $\mathbf{B}_{r_{2},i}$ have the following bounds: