Our previous neural network architectures (MLPs) treated inputs as independent entities. However, many real-world data types inherently possess a temporal or sequential dimension where the order of information matters significantly. Examples include:

• Text: The meaning of a word often depends on the words that precede it.
• Speech: Understanding a sentence requires processing sounds in sequence.
• Time Series Data: Stock prices, weather patterns, sensor readings, where future values depend on past values.
• Video: A sequence of images forming a coherent event.

Traditional MLPs are ill-suited for such tasks because they lack 'memory' of previous inputs in a sequence. This is where Recurrent Neural Networks (RNNs) come in.

The distinguishing feature of an RNN is its 'memory'. Unlike feedforward networks where information flows in one direction, RNNs have a hidden state that acts as a memory, capable of capturing information about the previous elements in the sequence. This hidden state is updated at each step of the sequence.

Conceptual Mechanism: Imagine a standard neuron that receives an input and produces an output. An RNN neuron (or 'recurrent cell') does this, but it also takes its own output from the previous time step as an additional input for the current time step.

• At each time step 't', the RNN takes two inputs:
• The current input from the sequence (xt).
• The hidden state (memory) from the previous time step (ht−1).
• It then combines these inputs, performs some calculations (including weights and biases, and an activation function), and produces two outputs:
• An output for the current time step (ot).
• An updated hidden state (ht) which is passed to the next time step.

This recurring connection (the feedback loop from hidden state to hidden state) allows the network to maintain information about past inputs, making it suitable for sequential data.

To better understand an RNN, we often 'unroll' it over time. This shows a series of standard neural network layers, where each layer represents a time step, and the hidden state from one layer feeds into the next. The crucial point is that the same weights and biases are reused across all time steps. This weight sharing is what enables RNNs to generalize across different positions in a sequence.

Despite their conceptual elegance, simple (vanilla) RNNs suffer from significant practical limitations, primarily due to the vanishing gradient problem (and sometimes exploding gradients) during backpropagation through time.

• Vanishing Gradients: As the network processes longer sequences, gradients for earlier time steps can become extremely small, effectively making it very difficult for the network to learn long-term dependencies (i.e., information from far back in the sequence has little influence on current predictions). The network 'forgets' distant past information.
• Exploding Gradients: Conversely, gradients can also become extremely large, leading to unstable training and large weight updates that prevent convergence. These problems made training vanilla RNNs on long sequences (like long sentences or entire paragraphs) very challenging. This led to the development of more sophisticated RNN architectures.

LSTMs, introduced by Hochreiter and Schmidhuber in 1997, are a special type of RNN specifically designed to address the vanishing gradient problem and effectively learn long-term dependencies. They do this by introducing a more complex internal structure called a 'cell state' and a system of 'gates' that control the flow of information.

Conceptual Overview of LSTM Gates: An LSTM cell has a central 'cell state' that runs straight through the entire sequence, acting like a conveyor belt of information. Information can be added to or removed from this cell state by a series of precisely controlled 'gates,' each implemented by a sigmoid neural network layer and a pointwise multiplication operation.
1. Forget Gate: This gate decides what information from the previous cell state should be 'forgotten' or discarded. It looks at the current input (xt) and the previous hidden state (ht−1) and outputs a number between 0 and 1 for each element in the cell state, where 1 means 'keep entirely' and 0 means 'forget entirely.'
2. Input Gate: This gate decides what new information from the current input should be stored in the cell state. It has two parts:
- A sigmoid layer (the 'input gate layer') decides which values to update.
- A tanh layer creates a vector of new candidate values that could be added to the cell state.
- These two parts combine to update the cell state.
3. Output Gate: This gate decides what part of the cell state should be outputted as the current hidden state (ht) and passed to the next time step. It uses a sigmoid layer to decide which parts of the cell state to filter, and then puts the cell state through a tanh function.

GRUs, introduced by Cho et al. in 2014, are a slightly simplified version of LSTMs. They combine the forget and input gates into a single 'update gate' and merge the cell state and hidden state.

Conceptual Overview of GRU Gates:
1. Update Gate: This gate determines how much of the previous hidden state to carry over to the current hidden state and how much of the new candidate hidden state to incorporate. It combines the functionality of the forget and input gates of an LSTM.
2. Reset Gate: This gate determines how much of the previous hidden state should be 'forgotten' or reset before computing the new candidate hidden state.

The choice between LSTMs and GRUs often depends on the specific task, dataset size, and computational resources. LSTMs are generally preferred for very long sequences or more complex tasks where precise memory control is critical. GRUs are a good alternative when computational efficiency is a higher priority or when the sequence dependencies are not extremely long. Both are significant advancements over vanilla RNNs.