Activation functions are critical non-linear components within a neural network neuron. They determine whether a neuron should be "activated" or "fired" based on the weighted sum of its inputs and bias. Without non-linear activation functions, a multi-layer neural network would simply be equivalent to a single-layer linear model, regardless of how many layers it has, because a series of linear transformations is still a linear transformation.

1. Sigmoid Function (Logistic Function):
2. Formula: sigma(z)=\frac{1}{1+e^{-z}}
3. Output Range: Maps any input value to a range between 0 and 1.
4. Shape: An 'S'-shaped curve.
5. Where Used: Historically popular in output layers for binary classification tasks (as it outputs probabilities) and in hidden layers.
6. Advantages:
   • Smooth gradient, which is good for backpropagation.
   • Output is squashed between 0 and 1, making it useful for probabilities.
7. Disadvantages:
   • Vanishing Gradient Problem: For very large positive or negative input values, the gradient (derivative) of the sigmoid function becomes extremely small (approaching zero). During backpropagation, these small gradients get multiplied across layers, causing gradients in earlier layers to effectively "vanish," slowing down or halting learning.
   • Outputs are not zero-centered: This can create issues during optimization, causing gradients to consistently be all positive or all negative, leading to "zigzagging" updates.
   • Computationally Expensive: The exponential operation is relatively slow.

1. Rectified Linear Unit (ReLU):
2. Formula: f(z)=max(0,z)
3. Output Range: Maps negative inputs to 0, and positive inputs to the input value itself.
4. Shape: A linear ramp.
5. Where Used: Most widely used activation function in hidden layers of deep neural networks.
6. Advantages:
   • Solves Vanishing Gradient (for positive inputs): For positive inputs, the gradient is always 1, preventing vanishing gradients and allowing faster training.
   • Computational Efficiency: Very simple and fast to compute (just a comparison and assignment).
   • Sparsity: Can lead to sparse activations, where many neurons output 0, which can be computationally efficient and potentially lead to more robust models.
7. Disadvantages:
   • Dying ReLU Problem: If a neuron's input is consistently negative, its output will be 0, and its gradient will also be 0. This means the neuron will never learn anything (it effectively "dies"). This issue can occur if learning rates are too high.
   • Not Zero-Centered: Similar to sigmoid, its output is not zero-centered.

1. Softmax Function:
2. Formula: For a vector of inputs z=[z_1,z_2,dots,z_K], the softmax function for the i-th element is:
   \[ softmax(z_i)=\frac{e^{z_i}}{\sum_{j=1}^{K}e^{z_j}} \]
3. Output Range: Transforms a vector of arbitrary real values into a probability distribution, where each element is between 0 and 1, and all elements sum up to 1.
4. Where Used: Exclusively used in the output layer for multi-class classification problems.
5. Advantages:
   • Provides interpretable probabilities for each class.
   • Forces the sum of probabilities to be 1, ensuring a valid probability distribution.
6. Disadvantages: Can also suffer from vanishing gradients for very large or very small inputs, similar to sigmoid, due to the exponential nature.

Why Non-linearity is Essential:
Without non-linear activation functions, a deep neural network, regardless of how many layers it has, would simply be equivalent to a single-layer linear model. This is because a composition of linear functions is always another linear function. Non-linearity introduced by activation functions allows neural networks to learn and model complex, non-linear relationships and patterns in data, which is fundamental to their power in deep learning.