Gradient Descent is the workhorse algorithm behind many machine learning models, especially for finding the optimal parameters. It's an iterative optimization algorithm used to find the minimum of a function. In the context of linear regression, this "function" is typically the cost function (e.g., Mean Squared Error), and we're looking for the values of our model's parameters (the β coefficients) that minimize this cost.

Imagine you're standing on a mountain peak, and your goal is to reach the lowest point (the valley). It's a foggy day, so you can't see the entire landscape, only the immediate slope around where you're standing. How would you find your way down? You'd likely take a small step in the direction that feels steepest downwards. Then, you'd re-evaluate the slope from your new position and take another step in the steepest downward direction. You'd repeat this process, taking small steps, always in the direction of the steepest descent, until you eventually reach the bottom.

The core idea is to iteratively adjust the parameters in the direction that most rapidly reduces the cost function. The general update rule for a parameter (let's use θj to represent any coefficient, like β0 or β1) is:

θj := θj − α ∂θj ∂ J(θ)

Let's break down this formula:
● θj: This is the specific model parameter (e.g., β0 or β1) that we are currently updating.
● :=: This means "assign" or "update." The parameter θj is updated to a new value.
● α (alpha): This is the Learning Rate. It's a crucial hyperparameter (a setting you choose before training).
○ Small Learning Rate: Means very small steps. The algorithm will take a long time to converge to the minimum, but it's less likely to overshoot.
○ Large Learning Rate: Means very large steps. The algorithm might converge quickly, but it could also overshoot the minimum repeatedly, oscillate around it, or even diverge entirely.
● J(θ): This represents the Cost Function (e.g., Mean Squared Error). Our goal is to minimize this function.
● ∂θj ∂ J(θ): This is the Partial Derivative of the cost function with respect to the parameter θj. It tells us the direction and steepness of the slope and indicates how much the cost changes if we slightly change θj.

There are three main flavors of Gradient Descent, distinguished by how much data they use to compute the gradient in each step:

3.2.1 Batch Gradient Descent
Intuition: Imagine our mountain walker has a magical drone that can instantly map the entire mountain from every angle. Before taking any step, the walker computes the exact steepest path considering the whole terrain. Then, they take that one perfectly calculated step.

Characteristics:
● Uses All Data: Batch Gradient Descent calculates the gradient of the cost function using all the training examples, making it computationally expensive but guaranteeing convergence for convex functions.
● Computationally Expensive: It processes the entire dataset for every update, which is slow for large datasets.
● Stable Updates: The gradient calculation is very accurate, leading to stable updates.

3.2.2 Stochastic Gradient Descent (SGD)
Intuition: Now, imagine our mountain walker is truly blindfolded and picks one pebble at random, feeling its immediate slope before moving.

Characteristics:
● Uses One Data Point: SDS updates parameters for each individual training example, making it faster for large datasets but leading to noisy updates.
● Noisy Updates: The path to the minimum is erratic, sometimes overshooting the actual minimum.

3.2.3 Mini-Batch Gradient Descent
Intuition: This is the most common and practical approach. Our mountain walker examines a small patch of the terrain (a "mini-batch" of pebbles).

Characteristics:
● Uses a Small Subset (Mini-Batch): It calculates updates using a small, randomly selected subset, striking a balance between speed and stability. It is commonly used in deep learning.