8.1.2. Activation Functions

Activation functions introduce non-linearity into the network.

Activation functions are crucial components of neural networks as they enable the model to learn complex patterns. Without activation functions, the neural network would only be able to represent linear relationships, severely limiting its capacity to solve real-world problems that often involve non-linearities. By introducing non-linearity, these functions help the network to understand and approximate various kinds of data.

Think of a light dimmer switch. If you could only turn the light on or off, you would only have two levels of brightness. But by using a dimmer, you can create a range of brightness levels, allowing for a more nuanced approach. Similarly, activation functions allow neural networks to adjust their output in a more flexible way, making them more effective.

Common activation functions include:

• Sigmoid: 1 Squashes input to range (0, 1) C3(x) = \frac{1}{1 + e^{-x}}
• Tanh: Output in range (-1, 1) $$tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$
• ReLU: Fast convergence, handles sparsity $$ReLU(x) = max(0, x)$$
• Leaky ReLU: Avoids dying neurons problem $$Leaky ReLU(x) = max(0.01x, x)$$
• Softmax: Used for multi-class classification $$softmax(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$$

There are several commonly used activation functions, each serving different purposes:

• Sigmoid: Specialized for binary classification, it maps input values to a range between 0 and 1. This is useful for models where outputs can be interpreted as probabilities.
• Tanh: Similar to sigmoid but stretches the output range from -1 to 1, making it centered around zero, which can sometimes result in faster convergence during training.
• ReLU (Rectified Linear Unit): This is a popular activation function for hidden layers. It replaces negative values with zero, allowing the network to maintain sparsity (many zero values) and usually improves performance significantly due to faster convergence.
• Leaky ReLU: A variation of ReLU that allows a small, non-zero gradient when the input is negative. This helps prevent neurons from becoming inactive or 'dying', which can happen with regular ReLU.
• Softmax: Typically applied in the output layer of models that must classify inputs into multiple categories. It converts raw scores (logits) into probabilities that sum to one, which can then be interpreted as the likelihood of each class.

Imagine you're sorting fruits based on color. The sigmoid function acts like a yes/no decision (red or not red), while tanh allows you to categorize fruit on a broader spectrum (red, yellow, green). ReLU acts like a light switch, letting through positive signals (like brightly colored fruits) while blocking the negative ones (dull or unwanted colors). Leaky ReLU allows a small amount of negative light to pass, ensuring that even if a signal is weak, it doesn’t completely get ignored.

--