On the Quality of the Initial Basin in Overspecified Neural Networks

Introduction

Deep learning (in the form of multi-layered artificial neural networks) has been tremendously successful in recent years, and advanced the state of the art across a range of difficult machine learning applications. Inspired by the structure of biological nervous systems, these predictors are usually composed of several layers of simple computational units (or neurons), parameterized by a set of weights, which can collectively express highly complex functions. Given a dataset of labeled examples, these networks are generally trained by minimizing the average of some loss function over the data, using a local search procedure such as stochastic gradient descent.

Although the expressiveness and statistical performance of such networks is relatively well-understood, it is a major open problem to understand the computational tractability of training such networks. Although these networks are trained successfully in practice, most theoretical results are negative. For example, it is known that finding the weights that best fit a given training set, even for very small networks, is NP-hard (Blum and Rivest, 1992). Even if we relax the problem by allowing improper learning or assuming the data is generated by a network, the problem remains worst-case hard (see e.g. (Livni et al., 2014) for a discussion of this and related results). This theory-practice gap is a prime motivation for our work.

In this paper, we study the geometric structure of the objective function associated with training such networks, namely the average loss over the training data as a function of the network parameters. We focus on plain-vanilla, feedforward networks which use the simple and popular ReLU activation function (see Sec. 2.1 for precise definitions), and losses convex in the network’s predictions, for example the squared loss and cross-entropy loss. The structure of the resulting objective function is poorly understood. Not surprisingly, it is complex, highly non-convex, and local search procedures are by no means guaranteed to converge to a global minimum. Moreover, it is known that even if the network is composed of a single neuron, the function may have exponentially many local minima (Auer et al., 1996). Furthermore, as we discuss later in the paper, the construction can be done such that the vast majority of these local minima are sub-optimal. Nevertheless, our goal in this work is to understand whether, perhaps under some conditions, the function has some geometric properties which may make it more favorable to optimization.

Before continuing, we emphasize that our observations are purely geometric in nature, independent of any particular optimization procedure. Moreover, we make no claim that these properties necessarily imply that a practical local search procedure, such as stochastic gradient descent, will converge to a good solution (although proving such a result could be an interesting direction for future work). Nevertheless, the properties we consider do seem indicative of the difficulty of the optimization problem, and we hope that our results can serve as a basis for further progress on this challenging research direction.

A recurring theme in our results is that such favorable properties can be shown to occur as the network size grows larger, perhaps larger than what would be needed to get good training error with unbounded computational power (hence the term overspecified networks). At first, this may seem counter-intuitive, as larger networks have more parameters, and training them involves apparently more complex optimization in a higher-dimensional space. However, higher dimensions also means more potential directions of descent, so perhaps the gradient descent procedures used in practice are more unlikely to get stuck in poor local minima and plateaus. Although difficult to formalize, this intuition accords with several recent empirical and theoretical evidence, which indicates that larger networks may indeed be easier to train (see (Livni et al., 2014) as well as (Choromanska et al., 2014; Dauphin et al., 2014; Bach, 2014)).

In the first part of our work (Sec. 3), we consider networks of arbitrary depth, where the weights are initialized at random using some standard initialization procedure. This corresponds to a random starting point in the parameter space. We then show that under some mild conditions on the loss function and the data set, as the network width increases, we are overwhelmingly likely to begin at a point from which there is a continuous, strictly monotonically decreasing path to a global minimumTo be precise, we prove a more general result, which implies a monotonic path to any objective value smaller than that of the initial point, as long as some mild conditions are met. See Thm. 1 in Sec. 3 for a precise formulation.. This means that although the objective function is non-convex, it is not “wildly” non-convex in the sense that the global minima are in isolated valleys which cannot be reached by descent procedures starting from random initialization. In other words, “crossing valleys” is not strictly necessary to reach a good solution (although again, we give no guarantee that this will happen for a specific algorithm such as stochastic gradient descent). We note that this accords well with recent empirical observations Goodfellow and Vinyals (2014), according to which the objective value of networks trained in practice indeed tends to decrease monotonically, as we move from the initialization point to the end point attained by the optimization algorithm. We also note that although we focus on plain-vanilla feed-forward networks, our analysis is potentially applicable to more general architectures, such as convolutional networks.

In the second part of our work (Sec. 4), we focus more specifically on two-layer networks with scalar-valued outputs. Although simpler than deeper networks, the associated optimization problem is still highly non-convex and exhibits similar worst-case computational difficulties. For such networks, we study a more fine-grained geometric property: We define a partition of the parameter space into convex regions (denoted here as basins), in each of which the objective function has a relatively simple, basin-like structure: Inside each such basin, every local minima of the objective function is global, all sublevel sets are connected, and in particular there is only a single connected set of minima, all global on that basin. We then consider the probability that a random initialization will land us at a basin with small minimal value. Specifically, we show that under various sets of conditions (such as low intrinsic data dimension, or a cluster structure), this event will occur with overwhelmingly high probability as the network size increases. As an interesting corollary, we show that the construction of (Auer et al., 1996), in which a single neuron network is overwhelmingly likely to be initialized at a bad basin, is actually surprisingly brittle to overspecification: If we replace the single neuron with a two-layer network comprised of just $\Omega(\log(d))$ neurons ($d$ being the data dimension), and use the same dataset, then with overwhelming probability, we will initialize at a basin with a globally optimal minimal value.

As before, we emphasize that these results are purely geometric, and do not imply that an actual gradient descent procedure will necessarily attain such good objective values. Nevertheless, we do consider a property such as high probability of initializing in a good basin as indicative of the optimization difficulty of the problem.

We now turn to discuss some related work. Perhaps the result most similar to ours appears in (Livni et al., 2014), where it is shown that quite generally, if the number of neurons in the penultimate layer is larger than the data size, then global optima are ubiquitous, and “most” starting points will lead to a global optimum upon optimizing the weights of the last layer. Independently, (Haeffele and Vidal, 2015) also provided results of a similar flavor, where sufficiently large networks compared to the data size and dimension do not suffer from local minima issues. However, these results involve huge networks, which will almost invariably overfit, whereas the results in our paper generally apply to networks of more moderate size. Another relevant work is (Choromanska et al., 2014), which also investigates the objective function of ReLU networks. That work differs from ours by assuming data sampled from a standard Gaussian distribution, and considering asymptotically large networks with a certain type of random connectivity. This allows the authors to use tools from the theory of spin-glass models, and obtain interesting results on the asymptotic distribution of the objective values associated with critical points. Other results along similar lines appear in (Dauphin et al., 2014). This is a worthy but rather different research direction than the one considered here, where we focus on theoretical investigation of non-asymptotic, finite-sized networks on fixed datasets, and consider different geometric properties of the objective function. Other works, such as (Arora et al., 2014; Andoni et al., 2014; Janzamin et al., 2015; Zhang et al., 2015) and some of the results in (Livni et al., 2014), study conditions under which certain types of neural networks can be efficiently learned. However, these either refer to networks quite different than standard ReLU networks, or focus on algorithms which avoid direct optimization of the objective function (often coupled with strong assumptions on the data distribution). In contrast, we focus on the geometry of the objective function, which is directly optimized by algorithms commonly used in practice. Finally, works such as (Bengio et al., 2005; Bach, 2014) study ways to convexify (or at least simplify) the optimization problem by re-parameterizing and lifting it to a higher dimensional space. Again, this involves changing the objective function rather than studying its properties.

Preliminaries and Notation

We use bold-faced letters to denote vectors, and capital letters to generally denote matrices. Given a natural number $k$, we let $\left[k\right]$ be shorthand for $\left\{1,\dots,k\right\}$.

For a vector $\mathbf{b}=\left(b_{1},\dots,b_{n}\right)$ and a matrix

and letting $\left[W\mathbf{x}+\mathbf{b}\right]_{+}$ be a shorthand for $\left(\left[\mathbf{w}_{1}^{\top}\mathbf{x}+b_{1}\right]_{+},\ldots,\left[\mathbf{w}_{n}^{\top}\mathbf{x}+b_{n}\right]_{+}\right)$, we can define a layer of $n$ neurons as

Finally, by denoting the output of the $i^{\text{th}}$ layer as $O_{i}$, we can define a network of arbitrary depth recursively by

where $W_{i},\mathbf{b}_{i}$ represent the matrix of weights and bias of the $i^{\text{th}}$ layer, respectively. Following a standard convention for multi-layer networks, the final layer $h$ is a purely linear function with no bias, i.e.

Define the depth of the network as the number of layers $h$, and denote the number of neurons $n_{i}$ in the $i^{\text{th}}$ layer as the size of the layer. We define the width of a network as $\max_{i\in\left[h\right]}n_{i}$.

We emphasize that in this paper, we focus on plain-vanilla networks, and in particular do not impose any constraints on the weights of each neuron (e.g. regularization, or having convolutional layers).

We define $\mathcal{W}$ to be the set of all network weights, which can be viewed as one long vector (its size of course depends on the size of the network considered). We will refer to the Euclidean space containing $\mathcal{W}$ as the parameter space.

2 Objective Function

In its simplest form, training a neural network corresponds to finding a combination of weights which minimizes the average loss over the training data. More formally, we define the objective function as

3 Basins

In Sec. 4, we will we consider a partition of the parameter space into convex regions, in each of which the objective function $L_{S}\left(\mathcal{W}\right)$ has a relatively simple basin-like form, and study the quality of the basin in which we initialize. In particular, we define a basin with respect to $L_{S}\left(\mathcal{W}\right)$ as a closed and convex subset of the parameter space, on which $L_{S}\left(\mathcal{W}\right)$ has connected sublevel sets, and where each local minimum is global. More formally, we have the following definition:

(Basin) A closed and convex subset $B$ of our parameter space is called a basin if the following conditions hold:

If $\mathcal{W}\in B$ is a local minimum of $L_{S}$ on $B$, then it is a global minimum of $L_{S}$ on $B$.

We define the basin value $\text{Bas}\left(B\right)$ of a basin $B$ as the minimal valueFor simplicity, we will assume this minimal value is actually attained at some point in the parameter space. Otherwise, one can refer to an attainable value arbitrarily close to it. attained:

Similarly, for a point $\mathcal{W}$ in the interior of a basin $B$ we define its basin value as the value of the basin to which it belongs:

In what follows, we consider basins with disjoint interiors, so the basin to which $\mathcal{W}$ belongs is always well-defined.

4 Initialization Scheme

As was mentioned in the introduction, we consider in this work questions such as the nature of the basin we initialize from, under some random initialization of the network weights. Rather than assuming a specific distribution, we will consider a general class of distributions which satisfy some mild independence and symmetry assumptions:

The initialization distribution of the network weights satisfies the following:

The weights of every neuron are initialized independently.

The vector of each neuron’s weights (including bias) is drawn from a spherically symmetric distribution supported on non-zero vectors.

Networks of Any Depth: Path to Global Minima

In this section, we establish the existence of a continuous path in the parameter space of multilayer networks (of any depth), which is strictly monotonically decreasing in the objective value, and can reach an arbitrarily small objective value, including the global minimum. More specifically, we show in Thm. 1 that if the loss is convex in the network’s predictions, and there exists some continuous path in the parameter space from the initial point $\mathcal{W}^{\left(0\right)}$ to a point with smaller objective value $\mathcal{W}^{\left(1\right)}$ (including possibly a global minimum, where the objective value along the path is not necessarily monotonic) which satisfies certain relatively mild conditions, then it is possible to find some other path from $\mathcal{W}^{\left(0\right)}$ to a point as good as $\mathcal{W}^{\left(1\right)}$, along which the objective value is strictly monotonically decreasing.

For the theorem to hold, we need to assume our starting point has a sufficiently large objective value. In Proposition 1 and Proposition 2, we prove that this will indeed occur with random initialization, with overwhelming probability. A different way to interpret this is that a significant probability mass of the surface of the objective function overlooks the global minimum. It should be noted that the path to the minimum might be difficult to find using local search procedures. Nevertheless, these results shed some light on the nature of the objective function, demonstrating that it is not “wildly” non-convex, in the sense that “crossing valleys” is not a must to reach a good solution, and accords with recent empirical evidence to this effect (Goodfellow and Vinyals, 2014).

For the results here, it would be convenient to re-write the objective function as $L(P(\mathcal{W}))$, where $\mathcal{W}$ is the vector of network parameters, $P(\mathcal{W})$ is an $m\times k$ matrix, which specifies the prediction for each of the $m$ training points (the prediction can be scalar valued, i.e. $k=1$, or vector-valued when $k>1$), and $L$ is the average loss over the training data. For example, for regression, a standard choice is the squared loss, in which case

Recall that although these losses are convex in the network’s predictions, $L(P(\mathcal{W}))$ is still generally non-convex in the network parameters $\mathcal{W}$. Also, we remind that due to the last layer being linear, multiplying its parameters by some scalar $c$ causes the output to change by $c$. Building on this simple observation, we have the following theorem.

For some $\epsilon>0$ and any $\lambda\in$, there exists some $c_{\lambda}\geq 0$ such that $L(c_{\lambda}\cdot P(\mathcal{W}^{\left(\lambda\right)}))\geq L(P(\mathcal{W}^{\left(0\right)}))+\epsilon$.

The initial point $\mathcal{W}^{\left(0\right)}$ satisfies $L(P(\mathcal{W}^{\left(0\right)}))>L(\mathbf{0})$.

Intuitively, this result stems from the linear dependence of the network’s output on the parameters of the last layer. Given the initial non-monotonic path $\mathcal{W}^{\left(\lambda\right)}$, we rescale the last layer’s parameters at each $\mathcal{W}^{\left(\lambda\right)}$ by some positive factor $c^{(\lambda)}$ depending on $\lambda$ (moving it closer or further from the origin), which changes its output and hence its objective value. We show it is possibly to do this rescaling, so that the rescaled path is continuous and has a monotonically decreasing objective value. In fact, although we focus here on ReLU networks, the theorem itself is quite general and holds even for networks with other activation functions. A formal proof and a more detailed intuition is provided in Subsection B.1.

The first condition in the theorem is satisfied by losses which get sufficiently large (as a function of the network predictions) sufficiently far away from the origin. In particular, it is generally satisfied by both the squared loss and the cross-entropy loss with softmax activations, assuming data points and initialization in general positionFor the squared loss, a sufficient condition is that for any $\lambda$, there is some data point on which the prediction of $N(\mathcal{W}^{\left(\lambda\right)})$ is non-zero. For the cross-entropy loss, a sufficient condition is that for any $\lambda$, there is some data point on which $N(\mathcal{W}^{\left(\lambda\right)})$ outputs an ‘incorrect’ prediction vector $\mathbf{p}$, in the sense that if $i$ is the correct label, then $p_{i}\notin\arg\max_{j}p_{j}$.. The second condition requires the random initialization to be such that the initialized network has worse objective value than the all-zeros predictor. However, it can be shown to hold with probability close to $1/2$ (over the network’s random initialization), for losses such as those discussed earlier:

If $L(\cdot)$ corresponds to the squared loss or cross-entropy loss with softmax activation, and the network parameters are initialized as described in Assumption 1, then

where $n_{h-1}$ is the number of neurons in the last layer before the output neurons.

This proposition (whose proof appears in appendix B.2) is a straightforward corollary of the following result, which can be applied to other losses as well:

Suppose the network parameters $\mathcal{W}^{\left(0\right)}$ are initialized randomly as described in Assumption 1. Suppose furthermore that $L(\cdot)$ is such that

for some $r>0$ (where the probability is with respect to $\mathcal{W}^{\left(0\right)}$). Then

Intuitively, the strict convexity property means that by initializing the neurons from a zero-mean distribution (such as a spherically symmetric one), we are likely to begin at a point with higher objective value than initializing at the mean of the distribution (corresponding to zero weights and zero predictions on all data points). A formal proof appears in Appendix B.3.

Two-layer ReLU Networks

We now turn to consider a more specific network architecture, namely two-layer networks with scalar output. While simpler than deeper architectures, two-layer networks still possess universal approximation capabilities (Cybenko, 1989), and encapsulate the challenge of optimizing a highly non-convex objective.

(Basin Partition) For any $A\in\left\{-1,+1\right\}^{n\times d}$ and $\mathbf{b}\in\left\{-1,+1\right\}^{n}$, define $B_{S}^{A,\mathbf{b}}$ as the topological closure of a set of the form

We will ignore $B_{S}^{A,\mathbf{b}}$ corresponding to empty sets, since these are irrelevant to our analysis.

For any $A\in\left\{-1,+1\right\}^{n\times d},~{}\mathbf{b}\in\left\{-1,+1\right\}^{n}$ such that $B_{S}^{A,\mathbf{b}}$ is non-empty, $B_{S}^{A,\mathbf{b}}$ is a basin as defined in Definition 1.

The reader is referred to Appendix A.1 for the proof of the lemma.

Note that Definition 2 refers to a partition of the parameter space into a finite number of convex polytopes. Recalling Assumption 1 on the initialization distribution (basically, that it is a Cartesian product of spherically-symmetric distributions), it is easy to verify that we will initialize in an interior of a basin with probability $1$. Therefore, we may assume that we always initialize in some unique basin.

We now focus on understanding when are we likely to initialize at a basin with a low minimal value (which we refer to as the basin value). We stress that this is a purely geometric argument on the structure on the objective function. In particular, even though every local minimum in a basin is also global on the basin, it does not necessarily entail that an optimization algorithm such as stochastic gradient descent will necessarily converge to the basin’s global minima (for example, it may drift to a different basin). However, we believe this type of geometric property is indicative of the optimization susceptibility of the objective function, and provides some useful insights on its structure.

We now turn to state a simple but key technical lemma, which will be used to prove the results presented later in this section. Moreover, this lemma also provides some insight into the geometry of the objective function for two-layer networks:

Let $N_{n}\left(W,\mathbf{v}\right)$ denote a two-layer network of size $n$, and let

be in the interior of some arbitrary basin. Then for any subset $I=\left(i_{1},\dots,i_{k}\right)\subseteq\left[n\right]$ we have

Where the right hand side is with respect to an architecture of size $k$.

This lemma captures in a way the power overspecification has in the context of two-layer networks: In terms of basin values, any initialization made using a network of width $n\geq k$ (i.e. with $n$ neurons in the first layer) is at least as good as if we had used only a width $k$ network. This is because in our definition of the basin partition, clamping the weights of any $n-k$ neurons to still keeps us in the same basin, while only increasing the minimal value we can obtain using the $k$ non-clamped neurons. Therefore, if we had only a $k$-width network to begin with, the corresponding basin value can only be larger. We refer the reader to Appendix A.2 for the proof of the lemma.

The training objective function of neural network is known to be highly non-convex, even for simple networks. A classic and stark illustration of this was provided in (Auer et al., 1996) who showed that even for a network comprised of a single neuron (with certain types of non-ReLU activation functions, and with or without bias), the objective function can contain a huge number of local minima (exponentially many in the input dimension). In Appendix C, we provide an extension of this result by proving that with a similar construction, and for a neuron with ReLU activation, not only is the number of local minima very large, but the probability of initializing at a basin with good local minimum (using the natural analogue of the basin partition from Definition 2 for a single neuron) is exponentially small in the dimension.

It is natural to study what happens to such a hardness construction under overspecification, which here means replacing a single neuron by a two-layer network of some width $n>1$, and training on the same dataset. Surprisingly, it turns out that in this case, the probability of reaching a sub-optimal basin decays exponentially in $n$ and becomes arbitrarily small already when $n=\Omega\left(\log\left(d\right)\right)$. Intuitively, this is because for such constructions, for each coordinate it is enough that one of the $n$ neurons in the first layer will have the corresponding weight initialized in the right basin. This will happen with overwhelming probability if $n$ is moderately large. More formally, we have the following theorem:

The reader is referred to Appendix B.4 for the full proof.

We note that $\alpha$ cannot be larger than the optimal value attained using a single neuron architecture. Also, we emphasize that the purpose of Thm. 2 is not to make a statement about neural networks for singleton datasets (which are not common in practice), but rather to demonstrate the brittleness of hardness constructions such as in (Auer et al., 1996) to overspecification, as more neurons are added to the first layer. This motivates us in further studying overspecification in the following subsections.

2 Data With Low Intrinsic Dimension

We now turn to provide a result, which demonstrates that for any dataset which is realizable using a two-layer network of a given width $n$ (i.e. $L_{S}\left(N_{n}\left(W,\mathbf{v}\right)\right)=0$ for some $\left(W,\mathbf{v}\right)$), the probability of initializing from a basin containing a good minimum increases as we add more neurons to the first layer, corresponding to the idea of overspecification. We note that this result holds without significant additional assumptions, but on the flip side, the number of neurons required to guarantee a constant success probability increases exponentially with the intrinsic dimension of the data ($\text{rank}\left(X\right)$, where $X$ is the data matrix whose rows are $\mathbf{x}_{1},\dots,\mathbf{x}_{m}$), so a magnitude of $\Omega\left(n^{\text{rank}\left(X\right)}\right)$ neurons is required. Thus, the result is only meaningful when the intrinsic dimension and $n$ are modest. In the next subsection, we provide results which require a more moderate amount of overspecification, under other assumptions.

where $B$ is some constant. For all $\epsilon>0$, if

and we initialize a two-layer, width $c\lceil\frac{n}{p_{\epsilon}}\rceil$ network (for some $c\geq 2$), using a distribution satisfying Assumption 1, then

The proof idea is that with a large enough amount of overspecification, with high probability, there will be a subset of the neurons in the first layer for which the signs of their outputs on the data and the signs of their weights in the output neuron will resemble those of $\left(W^{*},\mathbf{v}^{*}\right)$. Then, by using Lemma 2 we are able to argue that the initialization made in the remaining neurons does not degrade the value obtained in the aforementioned subset. We refer the reader to Appendix B.5 for the full proof.

3 Clustered or Full-rank Data

In this subsection, we will first show that when training on instances residing in high dimension $d$ (specifically, when the dimension satisfies $m\leq d$, where $m$ is the number of training examples), we initialize at a good basin with high probability. Building on this result, we show that even when $m>d$, we still initialize at a good basin with high probability, as long as the data is clustered into $k\leq d$ sufficiently small clusters.

Specifically, we begin by assuming that our data matrix $X$ satisfies $\text{rank}\left(X\right)=m$. We note that this immediately implies $m\leq d$. This refers to data of very high intrinsic dimension, which is in a sense the opposite regime to the one considered in the previous subsection (where the data was assumed to have low intrinsic dimension). Even though this regime might be strongly prone to overfitting, this allows us to investigate the surface area of the objective function effectively, while also serving as a base for the clustered data scenario that we will be studying in Thm. 5.

We now state our formal result for such datasets, which implies that under the rank assumption, a two-layer network of size $\mathcal{O}\left(\log\left(m\right)\right)$ is sufficient to initialize in a basin with a global minimum with overwhelming probability.

We refer the reader to Appendix B.6 for the full proof of the theorem.

As mentioned earlier, training on $m\leq d$ examples, without imposing any regularization, is prone to overfitting. Thus, to say something meaningful in the $m>d$ regime, we will consider an extension of the previous result, where instead of having fewer data points than dimensions $d$, we assume that the training instances are composed of $k\leq d$ relatively small clusters in general position. Intuitively, if the clusters are sufficiently small, the surface of the objective function will resemble that of having $k\leq d$ data points, and will have a similar favorable structure.

We also point out that in a similar manner to as we did in Thm. 3, the theorem statement assumes that the objective function refers to the average squared loss over the data. However, the proof does not rely on special properties of this loss, and it is possible to generalize it to other convex losses (perhaps with a somewhat different resulting bound).

$\exists\delta_{1},\dots,\delta_{k}>0$ s.t. for all $\mathbf{x}_{t}$, there is a unique $j\in[k]$ such that $\left\|\mathbf{c}_{j}-\mathbf{x}_{t}\right\|\leq\delta_{j}$.

$\forall j\in\left[k\right]\;\frac{\delta_{j}}{\left\|\mathbf{c}_{j}\right\|}\leq 2\sin\left(\frac{\sqrt{2\pi}}{16d\sqrt{d}}\right)$ and $\forall j\in\left[k\right]\text{ }\left\|\mathbf{c}_{j}\right\|\geq c$ for some $c>0$.

For some fixed $\gamma$, it holds that $\left|y_{t}-y_{t^{\prime}}\right|\leq\gamma\left\|\mathbf{x}_{t}-\mathbf{x}_{t^{\prime}}\right\|_{2}$ for any $t,t^{\prime}\in\left[m\right]$ such that $\mathbf{x}_{t},\mathbf{x}_{t^{\prime}}$ are in the same cluster.

Where the big $\mathcal{O}$ notation hides quadratic dependencies on $B,c^{-1},n,\sigma_{\min}^{-2}\left(C^{\top}\right),\sigma_{\max}\left(C^{\top}\right),\gamma,\left\|\hat{\mathbf{y}}\right\|_{2}$ (see the proof provided in Appendix B.7 for an explicit expression).

Note that $\delta$ measures how tight the clusters are, whereas $c,\sigma_{\max}\left(C^{\top}\right)$ and $\sigma_{\min}\left(C^{\top}\right)$ can be thought of as constants assuming the cluster centers are in general position. So, the theorem implies that for sufficiently tight clusters, with overwhelming probability, we will initialize from a basin containing a low-valued minimum, as long as the network size is $\Omega\left(\log\left(d\right)\right)$.

This research is supported in part by an FP7 Marie Curie CIG grant, Israel Science Foundation grant 425/13, and the Intel ICRI-CI Institute. We thank Lukasz Kaiser for pointing out a bug (as well as the fix) in an earlier version of the paper.

References

Appendix A Proofs of Basin Partition Properties

We will need the following three auxiliary lemmas.

Let $B$ be some basin as defined in Definition 2, and define $\mathbf{z}_{i}=v_{i}\mathbf{w}_{i}$. Then

is convex in $Z=\left(\mathbf{z}_{1},\dots,\mathbf{z}_{n}\right)$ on $B$.

If $v_{i}=0$ for some $i\in\left[n\right]$, then the $i^{\text{th}}$ neuron is canceled and we can linearly rescale $\mathbf{w}_{i}$ to $\mathbf{0}$, and then rescale $v_{i}$ to $b_{i}$, so we may assume without loss of generality that $v_{i}\neq 0$ for all $i\in\left[n\right]$. We have for all $\boldsymbol{\alpha}=\left(\alpha_{1},\dots,\alpha_{n}\right)\succ\mathbf{0}$,

Where we used the positive homogeneity of $\left[\cdot\right]_{+}$ in the last equality. So by linearly scaling $\boldsymbol{\alpha}^{\left(0\right)}=\left(1,\dots,1\right)$ to $\boldsymbol{\alpha}^{\left(1\right)}=\left(\left|v_{1}\right|,\dots,\left|v_{n}\right|\right)$, i.e. $\boldsymbol{\alpha}^{\left(\lambda\right)}=\left(1-\lambda+\lambda\left|v_{1}\right|,\dots,1-\lambda+\lambda\left|v_{n}\right|\right),\;\lambda\in\left[0,1\right]$, we obtain the desired path

$\left(\mathbf{w}_{i}^{\left(\lambda\right)},v_{i}^{\left(\lambda\right)}\right)\underset{\lambda\rightarrow 0_{+}}{\longrightarrow}\left(\mathbf{w}_{i},v_{i}\right)~{}~{}\forall i\in\left[n\right].$

$\left(\mathbf{w}_{1}^{\left(\lambda\right)},\dots,\mathbf{w}_{n}^{\left(\lambda\right)},v_{1}^{\left(\lambda\right)},\dots,v_{n}^{\left(\lambda\right)}\right)\in B_{S}^{A,\mathbf{b}}~{}~{}\forall\lambda\in\left(0,1\right).$

Can be shown using a straightforward computation.

Clearly, $B_{S}^{A,\mathbf{b}}$ is a closed set, and is convex as an intersection of halfspaces.

$B_{S}^{A,\mathbf{b}}$ has connected sublevel sets: Let $\left(W,\mathbf{v}\right),\left(W^{\prime},\mathbf{v}^{\prime}\right)\in B_{\leq\alpha}$. Using Lemma 4 we may assume without loss of generality that $\mathbf{v}=\mathbf{v}^{\prime}\in\left\{-1,+1\right\}^{n}$. By linearly interpolating $W,W^{\prime}$, i.e. by taking

we get a continuous path connecting $W,W^{\prime}$. This path remains in the same basin as a result of Lemma 5.3. Moreover, by Lemma 3, the objective is convex in $W$, so we get for all $\lambda\in\left[0,1\right]$

Any local minimum in $B_{S}^{A,\mathbf{b}}$ is global: Suppose $\left(W,\mathbf{v}\right)=\left(\mathbf{w}_{1},\dots,\mathbf{w}_{n},v_{1}\dots,v_{n}\right)$ is a local minimum in $B_{S}^{A,\mathbf{b}}$, let

Where the first transition comes from $\left(W,\mathbf{v}\right)$ being a local minimum and Lemma 5.2,5.3, the second and third from Lemma 5.1, and the fourth from Lemma 3.

A.2 Proof of Lemma 2

Appendix B Proofs of Main Theorems

Before delving into the proof of the theorem, we provide some intuition in the special case of the squared loss, where $L(P(\mathcal{W}))=\frac{1}{m}\sum_{t=1}^{m}(N(\mathcal{W})(\mathbf{x}_{t})-y_{t})^{2}$. Fix some $\lambda\in$, and consider the objective function along the ray in the parameter space, corresponding to multiplying the last layer weights in $\mathcal{W}^{\left(\lambda\right)}$ by some scalar $c\geq 0$. Since the output layer is linear, the objective function (as we vary $c$) will have the form

Thus, the objective function, as a parameter of $c$ (where $\mathcal{W}^{\left(\lambda\right)}$ is fixed) is just a quadratic function. Moreover, by the intermediate value theorem, as long as $N(\mathcal{W}^{\left(\lambda\right)})(\mathbf{x}_{t})$ is not for all $t$, then by picking different values of $c$, we can find points along the ray taking any value between $\frac{1}{m}\sum_{i=1}^{t}y_{t}^{2}$ (when $c=0$) and $\infty$ (as $c\rightarrow\infty$). Therefore, as long as we start from a point whose objective value is larger than $\frac{1}{m}\sum_{i=1}^{t}y_{t}^{2}$, we can re-scale each $\mathcal{W}^{\left(\lambda\right)}$ by some factor $c_{\gamma}$, so that the new path is continuous, as well as monotonically decreasing in value, remaining above $\frac{1}{m}\sum_{i=1}^{t}y_{t}^{2}$. When we reach the ray belonging to the endpoint $\mathcal{W}^{\left(1\right)}$ of the original path, we simply re-scale back towards $\mathcal{W}^{\left(1\right)}$, with the objective function continuing to decrease due to the convex quadratic form of the objective function along the ray.

We now turn to the formal proof in the general setting of Thm. 1. For technical reasons, we will extend the interval $\lambda\in$ to a strictly larger interval, and define certain quantities with respect to that larger interval. Specifically, for any $\lambda\in$, define

and note that it strictly monotonically decreases with $\lambda$, and satisfies the chain of inequalities

By assumption, for any $\lambda\in$, there exists some $c^{\left(\lambda\right)}$ such that $L(c^{\left(\lambda\right)}\cdot P(\mathcal{W}^{\left(\lambda\right)}))\geq L(P(\mathcal{W}^{\left(0\right)}))+\epsilon$. Since $L(P(\mathcal{W}^{\left(0\right)}))+\epsilon>v^{\left(\lambda\right)}$ for any $\lambda\in$, it follows that for any such $\lambda$,

where $\text{clip}(\lambda)=\min\{1,\max\{0,\lambda\}\}$ denote clipping of $\lambda$ to the interval $$. On the other hand, for any$\lambda\in$,

Since $L$ is convex and real-valued, it is continuous, hence $L(c\cdot P(\mathcal{W}^{\text{clip}(\lambda)}))$ is convex and continuous in $c$. Combining this with Eq. (2) and Eq. (3), it follows from the intermediate value theorem that

which cannot be satisfied by a convex function $f$.

B.2 Proof of Proposition 1

It is enough to verify that for both losses, proposition 2 holds with $r=1$.

For the squared loss, if $P(\mathcal{W}^{\left(0\right)})\neq\mathbf{0}$, then consider the first training example $(\mathbf{x}_{i},y_{i})$ for which $P(\mathcal{W}^{\left(0\right)})(\mathbf{x}_{i})\neq\mathbf{0}$. In that case, it is easily verified that $(c\cdot P(\mathcal{W}^{\left(0\right)})(\mathbf{x}_{i})-y_{i})^{2}$ is strictly convex in $c$ (for any $c$), and therefore $L(c\cdot P(\mathcal{W}^{\left(0\right)}))=\frac{1}{m}\sum_{i=1}^{m}(c\cdot P(\mathcal{W}^{\left(0\right)})(\mathbf{x}_{i})-y_{i})^{2}$ is also strictly convex, as an average of convex functions where at least one of them is strictly convex. Therefore, strict convexity holds with probability $r=1$.

For the cross-entropy loss, it is enough to consider the first training example on which the prediction vector $\mathbf{p}$ of $P(\mathcal{W}^{\left(\lambda\right)})$ is non-zero, and show strict convexity on that example with probability $1$. Since the loss on other examples are convex as well, we get overall strict convexity with probability $1$ as required. Specifically, we need to show strict convexity in $c$ of the function

where $j_{i}$ is the correct class. To do so, consider the function $f(\mathbf{p})=\log(\sum_{j}\exp(p_{j}))$. A straightforward calculation reveals that its Hessian equals

so the second derivative of the function in Eq. (6) w.r.t. $c$ at some value $c$ equals

We now argue that this is strictly positive, unless $\mathbf{p}$ is a constant vector $p_{1}=p_{2}=\ldots=p_{k}$, in which case the function in Eq. (6) is indeed strictly convex. To see this, note that the Hessian of $f$ is a rank-1 perturbation of the $k\times k$ positive definite matrix $\text{diag}(\mathbf{q})$, so its rank is at least $k-1$. Thus, there is only a $1$-dimensional subspace of vectors $\mathbf{v}$, for which $\mathbf{v}^{\top}(\text{diag}(\mathbf{q})-\mathbf{q}\mathbf{q}^{\top})\mathbf{v}=0$, which can be verified to be exactly the subspace of constant vectors. Thus, Eq. (7) is positive unless $\mathbf{p}$ is a constant vector.

To finish the proof for the cross-entropy loss, it remains to show that the probability that $\mathbf{p}$ is non-constant (conditioned on $P(\mathcal{W}^{\left(0\right)})\neq\mathbf{0})$ is $1$. To simplify the notation, let $NZ$ be the event that $P(\mathcal{W}^{\left(0\right)})\neq\mathbf{0}$, let $P$ be the event that conditioned on $NZ$, $\mathbf{p}$ (the first non-zero prediction vector over the training examples) is also non-constant. Also, let $V$ be the event that conditioned on $NZ$, then for the same training example as $\mathbf{p}$, the input vector to the output neurons is non-zero. Then it holds that

B.3 Proof of Proposition 2

B.4 Proof of Thm. 2

and observe the objective value on $S_{j}^{+}$ satisfies for all $j\in\left[d\right]$,

The probability of this condition not to hold for a single neuron is at most $\frac{3}{4}$.

The probability of this condition not to hold for all neurons (since by Assumption 1 all neurons are independent) is at most $\left(\frac{3}{4}\right)^{n}$.

By using the union bound, the probability that exists some $S_{j}^{\pm}$ such that no neuron can obtain the minimal value over it is at most

We conclude that when initializing $\left(W,\mathbf{v}\right)$ using a distribution satisfying Assumption 1 then

B.5 Proof of Thm. 3

We will need the following two auxiliary lemmas:

$N_{n}\left(\mathbf{w}_{1},\dots,\mathbf{w}_{n},\mathbf{v}\right)\left(\mathbf{x}\right)$ is $\left(\left|v_{i}\right|\cdot\left\|\mathbf{x}\right\|\right)$-Lipschitz in each $\mathbf{w}_{i}$.

We leave this lemma without proof, and note that it is immediate from the definition of $N_{n}$.

The following lemma provides a lower bound on the probability that the $i^{\text{th}}$ neuron is initialized from a region with a point of distance at most $\delta$ from $\mathbf{w}^{*}_{i}$.

Let $\delta>0$, let $\left(W^{*},\mathbf{v}^{*}\right)$ satisfying $\left\|\mathbf{w}^{*}_{i}\right\|_{2}=1,~{}\forall i\in\left[n\right]$, and let $\left(W,\mathbf{v}\right)$ be a point on an origin-centered sphere chosen uniformly at random. Then $\forall i\in\left[n\right]$

Before turning to prove the lemma, we first prove the following auxiliary claim.

This claim suffices for proving a weaker version of Thm. 3 where $\text{rank}\left(X\right)$ is replaced with $d$. However, utilizing a simple observation on the structure of the basin partition allows us to prove Lemma 7 which strengthens the result.

The surface area of a hyperspherical cap of radius $\theta$ is given by the formula: ([Li, 2011])

where the last inequality holds for all $\theta\in\left[0,\pi\right]$. Since $f\left(0\right)=0$ we have $\forall\theta\in\left[0,\pi\right]$ that $\int_{0}^{\theta}\left(\sin^{d-2}\xi d\xi\right)\geq\frac{1}{d-1}\sin^{d-1}\theta$.

Using the identities $\sin\left(\arcsin x\right)=x$, $\cos\left(\arcsin x\right)=\sqrt{1-x^{2}}$ and $\sin 2x=2\sin x\cdot\cos x$, we have

Finally, $\frac{\omega_{d-2}}{\omega_{d-1}}$ can be shown to be monotonically increasing for all $d\geq 2$, so $\frac{\omega_{d-2}}{\omega_{d-1}}\geq\frac{\omega_{0}}{\omega_{1}}=\frac{1}{\pi}$, thus yielding

which concludes the proof of the claim. ∎

Let $U=\text{span}\left(\mathbf{x}_{1},\dots,\mathbf{x}_{m}\right)$, and define $T\left(\mathbf{x}\right)\coloneqq\frac{\mathbf{u}}{\left\|\mathbf{u}\right\|_{2}}$ where $\mathbf{x}=\mathbf{u}+\mathbf{u}^{\perp}$ for $\mathbf{u}\in U,\mathbf{u}^{\perp}\in U^{\perp}$. First, we observe that for any initialization of $\left(W,\mathbf{v}\right)$ , that $\left(W,\mathbf{v}\right)$ and $\left(T\left(W\right),\mathbf{v}\right)$ where $T\left(W\right)\coloneqq\left(T\left(\mathbf{w}_{1}\right),\dots,T\left(\mathbf{w}_{n}\right)\right)$ both belong to the same basin, since $\forall i\in\left[n\right],\forall t\in\left[m\right]$

Thus both $W,T\left(W\right)$ belong to the same basin, achieving the same minimal value. Since any rotation $\Theta$ under which $U^{\perp}$ is invariant commutes with $T$, we have for any measurable set $A\subseteq U$

where $\sigma\left(k\right)$ denotes the $k$-dimensional Lebesgue measure. So initializing uniformly on an origin-centered sphere of dimension $d$ is equivalent to initializing uniformly on an origin-centered sphere of dimension $\text{rank}\left(X\right)$ in the sense of the region we initialize from. We complete the proof by invoking Claim 1 with respect to a $\left(\text{rank}\left(X\right)\right)$-dimensional sphere. ∎

We first argue that since our initialization distribution satisfies Assumption 1, we may rescale each neuron once initialized to the unit sphere. This is justified since a linear rescaling of the weight of each neuron does not change the basin we initialized from, so the basin value remains the same. For this reason, we assume without loss of generality the distribution where each neuron is distributed uniformly and independently on the unit sphere. Define

Using the positive homogeneity of the ReLU, we can rescale each $v_{i}^{*}$ to satisfy $\left|v_{i}^{*}\right|=1~{}\forall i\in\left[n\right]$, and rescale $\mathbf{w}_{i}^{*}$ accordingly, so we may also assume $\left|v_{i}^{*}\right|=1,\left\|\mathbf{w}_{i}^{*}\right\|\leq B~{}\forall i\in\left[n\right]$. From Lemma 7 we have

Since the two events are independent, we have that both occur w.p. at least $p_{\epsilon}$. Also, this event is independent for each neuron, so we have w.p. at least $p_{\epsilon}$ for each neuron to initialize ‘close‘ enough to $\left(\mathbf{w}_{i}^{*},v_{i}^{*}\right)$. In this sense, we can lower bound the number of good initializations from below using $Z\sim B\left(N,p_{\epsilon}\right)$, where $B\left(N,p\right)$ is the binomial distribution. By using Chernoff‘s bound we bound the tail of $Z$ as follows

Let $i_{1},\dots,i_{n}$ be the indices of the well initialized neurons, and let

We compute the value of the basin corresponding to these neurons as follows:

where the second inequality in the triangle inequality and the third inequality is from Lemma 6. We now finish the proof by invoking Lemma 2 to conclude

B.6 Proof of Thm. 4

we want to show that for well chosen values of $a_{i,t^{\prime}}$, $\left(W^{\prime},\mathbf{v}^{\prime}\right)$ belongs to the same basin as $\left(W,\mathbf{v}\right)$, and achieves the desired prediction $\left(\bar{y}_{1},\dots,\bar{y}_{m}\right)$ over a certain subset of $\left(\mathbf{x}_{1},\dots,\mathbf{x}_{m}\right)$, while achieving a prediction of over the rest of the sample instances, effectively predicting the subset without affecting the prediction over the rest of the sample. By combining enough neurons in this manner, we are able to obtain the minimal objective value over the data. Namely, an objective value of $\alpha$. Define the vector $\mathbf{y}_{i}^{\prime}=\left(y_{i,1}^{\prime},\ldots,y_{i,m}^{\prime}\right)^{\top}$, where

and choose $\mathbf{a}_{i}=\left(a_{i,1},\ldots,a_{i,m}\right)^{\top}$ such that the equality

is of rank $m$, and therefore $\mathbf{a}_{i}$ exists and is well-defined.

Assuming that for any $t\in\left[m\right]$ there exists some neuron $i$ such that $\left\langle\mathbf{w}_{i},\mathbf{x}_{t}\right\rangle>0,\;v_{i}\cdot\bar{y}_{t}\geq 0$ (We will later analyze the probability of this actually happening), we compute the prediction of our network with weights $W^{\prime}=\left(\mathbf{w}_{1}^{\prime},\dots,\mathbf{w}_{n}^{\prime}\right)$ on $\mathbf{x}_{t}$:

Where the last equality comes from our assumption that there exists some neuron $i$ s.t. $\left\langle\mathbf{w}_{i},\mathbf{x}_{t}\right\rangle>0,\;v_{i}\cdot\bar{y}_{t}\geq 0$, and from the definition of $y_{i,t}^{\prime}$ which asserts that at most a single neuron will predict $\mathbf{x}_{t}$. Thus, we have

To put this result in different words, if $\mathbf{x}_{t}$ is positive on the hyperplane induced by $\mathbf{w}_{i}$ and if $v_{i}$ has the same sign as $\bar{y}_{t}$, then $\mathbf{w}_{i}^{\prime}$ predicts $\mathbf{x}_{t}$ correctly, given that $\mathbf{x}_{t}$ was not previously predicted by a neuron $\mathbf{w}_{j}^{\prime}$ where $j<i$.

Finally, we define the event $A_{i}^{t}\coloneqq\left\langle\mathbf{w}_{i},\mathbf{x}_{t}\right\rangle>0,\;v_{i}\cdot\bar{y}_{t}\geq 0$, i.e. the $i^{\text{th}}$ neuron is able to predict $\mathbf{x}_{t}$ correctly. Since $v_{i},\mathbf{w}_{i}$ are independent, and since $\mathbf{w}_{i}$ is drawn from a spherically symmetric distribution for all $i\in\left[n\right]$, we have

Using the union bound on $\bigcap_{i=1}^{n}\overline{A_{i}^{t}}$ for $t=1,\dots,m$ we get

Thus, the probability of initializing from a basin achieving a global minimum with value $\alpha$ is at least

B.7 Proof of Thm. 5

The idea behind the proof is comprised of two parts. The first is that by predicting the clusters’ centers well, we are able to obtain a good objective value over the data. The second is that the basin partition of the clustered data is similar to the basin partition of the clusters’ centers. So by approximating a good solution for the clusters’ centers, we are able to reach a good objective value.

To approximate a good solution for the clusters’ centers, we need to initialize from a basin where such an approximation exists. Note that if $\delta=0$, then the result will hold as a corollary of Thm. 4. Alternatively, if $\delta$ is small enough, then we would expect such an approximation to exist in the basins comprised of non-noisy regions, as these vary slightly when $\delta$ is small. Therefore, we would like to assert that we initialize from these basins to guarantee the existence of a good solution.

Before delving into the proof of Thm. 5, we first prove two auxiliary lemmas (Lemma 8 and Lemma 9). The following lemma provides an upper bound on initializing a single neuron from a noisy region, for distributions satisfying Assumption 1.

Define the set of noisy regions with respect to the $j^{\text{th}}$ cluster,

Where $\sigma_{d-1}$ is the $\left(d-1\right)$-dimensional Lebesgue measure, and $\omega_{d-1}$ is the surface area of the $d$-dimensional unit sphere.

To prove the lemma we will need two auxiliary claims.

Where the same argument works for $\mathbf{x}\in\mathsf{S}\left(-\mathbf{c}_{j},\frac{\pi}{2}-2\arcsin\frac{\delta_{j}}{2\left\|\mathbf{c}_{j}\right\|}\right)$ and $-P_{j}$. ∎

Consider the function $f\left(\theta\right)=\left(\int_{0}^{\frac{\pi}{2}-\theta}\sin^{d-2}\xi d\xi\right)-\left(\frac{\omega_{d-1}}{2\omega_{d-2}}-\theta\right)$, it is monotonically increasing in $\left[0,\infty\right)$ since

And since $f\left(0\right)=0$ we have $\forall\theta\in\left[0,\infty\right)$ that $\int_{0}^{\frac{\pi}{2}-\theta}\sin^{d-2}\xi d\xi\geq\frac{\omega_{d-1}}{2\omega_{d-2}}-\theta$. ∎

Using Claims 2 and 3, Eq. (10) and the fact that $\forall d\geq 2\;\frac{\omega_{d-1}}{\omega_{d}}\leq\sqrt{\frac{d}{2\pi}}$ ([Leopardi, 2007], Lemma 2.3.20), we have the following:

Before proving the lemma, we state and prove the following two auxiliary claims.

$N_{n}\left(\mathbf{w}_{1},\dots,\mathbf{w}_{n},\mathbf{v}\right)\left(\mathbf{x}\right)$ is $\left(\sum_{i=1}^{n}\left|v_{i}\right|\cdot\left\|\mathbf{w}_{i}\right\|\right)$-Lipschitz in $\mathbf{x}$.

The proof of this claim follows the same idea behind Lemma 6, and is therefore omitted.

where the last inequality comes from $y_{t},\hat{y}_{t}$ being the target values of points belonging to a ball of diameter at most $2\delta$ and the target values being $\gamma$-Lipschitz.

Equipped with the above lemmas, we are now ready to prove Thm. 5.

Using Lemma 8, we have for all $j\in\left[k\right]$

We stress that by using Lemma 2, for the purpose of analyzing the objective value, we can ignore initializations made from noisy regions, as we may just consider the neurons that were properly initialized. By our assumption that the clusters’ centers are in general position, namely that the matrix $C$ with rows $\mathbf{c}_{1},\dots,\mathbf{c}_{k}$ satisfies $\sigma_{\min}\left(C^{\top}\right)>0$, we have that it is in particular of rank $k$, and the conditions in Lemma 9 are met, so we compute

Thus we conclude that when $\left(W,\mathbf{v}\right)$ is initialized using a distribution satisfying Assumption 1, we have

Appendix C Poor Basin Structure for Single Neurons

In this appendix, we prove a hardness result for initializing ReLU single neuron nets with convex losses from a basin (as will shortly be defined for the single neuron architecture context) with a good basin value, and then provide an explicit construction for the squared loss.

Building on the work of [Auer et al., 1996], we provide a construction of a dataset which results in exponentially many poor local minima in the dimension. Moreover, we provide in subsection Appendix 3 an explicit construction for the squared loss. The results extend those of [Auer et al., 1996] by showing that they hold for a single neuron with the ReLU activation function (for which the technical conditions assumed in [Auer et al., 1996] do not apply).

From an optimization point of view, having exponentially many local minima is not necessarily problematic as many of which may obtain good objective values. However, following our initialization scheme throughout this work, we modify the result obtained in [Auer et al., 1996] to satisfy that when the weight vector of the neuron is initialized from a distribution satisfying Assumption 1, then the distribution of the minimal value in the basin we initialize from is strongly concentrated around a sub-optimal value as the dimension increases. More formally, we have the following Theorem.

Where $\mathbf{w}$ is initialized according to Assumption 1.

In other words, we have exponentially many local minima, where the probability of initializing from a sub-optimal basin converges exponentially fast (in the dimension) to $1$, yet there exists a solution which obtains a value of $\epsilon$.

We now extend our sample to be $d$-dimensional in a similar manner as did the authors in [Auer et al., 1996] as follows: For $t=1,2$ and $j\in\left[d\right]$, we use the mapping $x_{t}\left(j\right)\mapsto\left(0,\dots,0,x_{t},0,\dots,0\right)$ where the non-zero coordinate is the $j^{\text{th}}$ coordinate. It is straightforward to show that the partial derivative $\frac{\partial}{\partial w_{j}}L_{S}$ is for $x_{t}\left(k\right)$ with $j\neq k$, so the geometry of the surface of the objective function $L_{S}$ is independent for each coordinate. Now, every Cartesian product of local minima in the one-dimensional setting form a $d$-dimensional local minimum. Since we have exactly two local minima, a good and another bad one in each coordinate, this combines into $2^{d}$ local minima, where each minimum’s value would be the average of the one-dimensional minima forming it. Note that the combination of all good minima forms the global minimum with value $\epsilon$. Following standard convention, we say that the data in this case is $\epsilon$-realizable using a single neuron architecture. We stress that an important property of this initialization scheme is that the signs of the coordinates of the initialization point is uniformly distributed on the Boolean cube, as it implies that on each coordinate, independently, we have a probability $0.5$ of reaching a bad basin, hence the number of bad basins we initialize from is distributed according to a Binomial distribution $B\left(d,0.5\right)$. Letting $c=\frac{L_{0}}{2}$, we have from Chernoff‘s bound that

which concludes the proof of the theorem. ∎

C.2 An Explicit Construction With the Squared Loss

We illustrate a specific construction of Thm. 6, for ReLU paired with the squared loss.

Given $\epsilon>0$, consider the following sample:

are both local minima, and thus $S$ is $\epsilon$-realizable. As evident in Fig. 2 and Fig. 3, if we are using a distribution corresponding to Assumption 1, then we have a $50\%$ chance to initialize from the bad basin. Extending the sample into a $d$-dimensional one as we did in Thm. 6, we have an $\epsilon$-realizable dataset $S$ with $2^{d}$ local minima. Furthermore, we have that