Weight initialization

LeCun initialization

LeCun initialization, popularized in (LeCun et al., 1998),^[5] is designed to preserve the variance of neural activations during the forward pass.

It samples each entry in ${\displaystyle W^{(l)}}$ independently from a distribution with mean 0 and variance ${\displaystyle 1/n_{l-1}}$. For example, if the distribution is a continuous uniform distribution, then the distribution is ${\displaystyle {\mathcal {U}}(\pm {\sqrt {3/n_{l-1}}})}$.

Glorot initialization

Glorot initialization (or Xavier initialization) was proposed by Xavier Glorot and Yoshua Bengio.^[6] It was designed as a compromise between two goals: to preserve activation variance during the forward pass and to preserve gradient variance during the backward pass.

For uniform initialization, it samples each entry in ${\displaystyle W^{(l)}}$ independently and identically from ${\displaystyle {\mathcal {U}}(\pm {\sqrt {6/(n_{l+1}+n_{l-1})}})}$. In the context, ${\displaystyle n_{l-1}}$ is also called the "fan-in", and ${\displaystyle n_{l+1}}$ the "fan-out". When the fan-in and fan-out are equal, then Glorot initialization is the same as LeCun initialization.

He initialization

As Glorot initialization performs poorly for ReLU activation,^[7] He initialization (or Kaiming initialization) was proposed by Kaiming He et al.^[8] for networks with ReLU activation. It samples each entry in ${\displaystyle W^{(l)}}$ from ${\displaystyle {\mathcal {N}}(0,2/n_{l-1})}$.

Orthogonal initialization

(Saxe et al. 2013)^[9] proposed orthogonal initialization: initializing weight matrices as uniformly random (according to the Haar measure) semi-orthogonal matrices, multiplied by a factor that depends on the activation function of the layer. It was designed so that if one initializes a deep linear network this way, then its training time until convergence is independent of depth.^[10]

Sampling a uniformly random semi-orthogonal matrix can be done by initializing ${\displaystyle X}$ by IID sampling its entries from a standard normal distribution, then calculate ${\displaystyle \left(XX^{\top }\right)^{-1/2}X}$ or its transpose, depending on whether ${\displaystyle X}$ is tall or wide.^[11]

For CNN kernels with odd widths and heights, orthogonal initialization is done this way: initialize the central point by a semi-orthogonal matrix, and fill the other entries with zero. As an illustration, a kernel ${\displaystyle K}$ of shape ${\displaystyle 3\times 3\times c\times c'}$ is initialized by filling ${\displaystyle K[2,2,:,:]}$ with the entries of a random semi-orthogonal matrix of shape ${\displaystyle c\times c'}$, and the other entries with zero. (Balduzzi et al., 2017)^[12] used it with stride 1 and zero-padding. This is sometimes called the Orthogonal Delta initialization.^[11][13]

Related to this approach, unitary initialization proposes to parameterize the weight matrices to be unitary matrices, with the result that at initialization they are random unitary matrices (and throughout training, they remain unitary). This is found to improve long-sequence modelling in LSTM.^[14][15]

Orthogonal initialization has been generalized to layer-sequential unit-variance (LSUV) initialization. It is a data-dependent initialization method, and can be used in convolutional neural networks. It first initializes weights of each convolution or fully connected layer with orthonormal matrices. Then, proceeding from the first to the last layer, it runs a forward pass on a random minibatch, and divides the layer's weights by the standard deviation of its output, so that its output has variance approximately 1.^[16][17]

Fixup initialization

In 2015, the introduction of residual connections allowed very deep neural networks to be trained, much deeper than the ~20 layers of the previous state of the art (such as the VGG-19). Residual connections gave rise to their own weight initialization problems and strategies. These are sometimes called "normalization-free" methods, since using residual connection could stabilize the training of a deep neural network so much that normalizations become unnecessary.

Fixup initialization is designed specifically for networks with residual connections and without batch normalization, as follows:^[18]

1. Initialize the classification layer and the last layer of each residual branch to 0.
2. Initialize every other layer using a standard method (such as He initialization), and scale only the weight layers inside residual branches by ${\displaystyle L^{-{\frac {1}{2m-2}}}}$.
3. Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each convolution, linear, and element-wise activation layer.

Similarly, T-Fixup initialization is designed for Transformers without layer normalization.^[19]: 9

Others

Instead of initializing all weights with random values on the order of ${\displaystyle O(1/{\sqrt {n}})}$, sparse initialization initialized only a small subset of the weights with larger random values, and the other weights zero, so that the total variance is still on the order of ${\displaystyle O(1)}$.^[20]

Random walk initialization was designed for MLP so that during backpropagation, the L2 norm of gradient at each layer performs an unbiased random walk as one moves from the last layer to the first.^[21]

Looks linear initialization was designed to allow the neural network to behave like a deep linear network at initialization, since ${\displaystyle W\;\mathrm {ReLU} (x)-W\;\mathrm {ReLU} (-x)=Wx}$. It initializes a matrix ${\displaystyle W}$ of shape ${\displaystyle \mathbb {R} ^{{\frac {n}{2}}\times m}}$ by any method, such as orthogonal initialization, then let the ${\displaystyle \mathbb {R} ^{n\times m}}$ weight matrix to be the concatenation of ${\displaystyle W,-W}$.^[22]