Self-Normalizing Networks

Tutorials and implementations for Self-Normalizing Networks (SNNs) as suggested by Klambauer et al. (arXiv pre-print).

The repository contains TensorFlow/Keras scripts and PyTorch notebooks for demonstrating the main SNN recipe:

• SELU activations
• LeCun normal initialization
• AlphaDropout when dropout is used

Environment

The current examples use TensorFlow 2.x/Keras and PyTorch. Create the Conda environment from the root environment file:

conda env create -f environment.yml
conda activate snn

The environment is intended for current TensorFlow and PyTorch versions. On Linux/NVIDIA systems, TensorFlow GPU dependencies are installed via the tensorflow[and-cuda] pip extra.

Implementation Notes

TensorFlow/Keras

Keras provides SELU, LeCun normal initialization, and AlphaDropout:

• tf.keras.activations.selu
• tf.keras.initializers.LecunNormal
• tf.keras.layers.AlphaDropout

PyTorch

PyTorch provides SELU and AlphaDropout:

• torch.nn.SELU
• torch.nn.AlphaDropout

For SELU networks, initialize linear and convolutional layers with:

torch.nn.init.kaiming_normal_(weight, mode="fan_in", nonlinearity="linear")

This corresponds to LeCun normal initialization because fan_in is used with a gain of 1.

Tutorials

TensorFlow 2.x / Keras

• Multilayer Perceptron on MNIST (python script)
• Convolutional Neural Network on MNIST (python script)
• Convolutional Neural Network on CIFAR10 (python script)

PyTorch

• Multilayer Perceptron on MNIST (notebook)
• Convolutional Neural Network on MNIST (notebook)
• Convolutional Neural Network on CIFAR10 (notebook)

Further Material

Figure 1

Notebooks and code to produce Figure 1 are provided in figure1. This material builds on top of the biutils package.

Calculations and Numeric Checks

Mathematica notebooks for calculations and numeric checks of the theorems are provided here:

• Mathematica notebook
• Mathematica PDF

UCI, Tox21, and HTRU2 Data Sets

• UCI
• Tox21
• HTRU2