9.6. Deep Learning in NLP

• Useful for sequential text data, but suffers from vanishing gradient problems.

Recurrent Neural Networks (RNN) are a type of artificial neural network designed to recognize patterns in sequences of data, such as text. Unlike traditional neural networks, which assume that the inputs to their models are independent, RNNs are particularly useful for tasks where context and order matter, as they keep a 'memory' of previous inputs. However, RNNs can face a major issue known as the 'vanishing gradient problem.' This occurs when the gradients (used to train the model) become too small, effectively halting training progress for long-term dependencies. This means that while RNNs can process sequential data, they struggle to connect information over longer sequences.

Imagine a person reading a sentence. If they need to recall what was said ten words earlier, they'll likely struggle to remember it if the sentence is too long. Similarly, RNNs find it hard to remember information from earlier in a long sequence of text, which limits their effectiveness for tasks involving longer contexts.

• Overcomes RNN limitations, better at long-term dependencies.

Long Short-Term Memory (LSTM) networks are a special type of RNN specifically designed to avoid the vanishing gradient problem. They achieve this through a unique structure that includes gates. These gates determine how much information should be remembered and forgotten over time, allowing LSTMs to retain relevant information for longer sequences. Gated Recurrent Units (GRU) are a simplified version of LSTMs that combine the forget and input gates into a single update gate, making them faster to train while retaining similar benefits. Together, LSTMs and GRUs are widely used for applications involving sequential data, such as time series prediction and language modeling.

Think of LSTMs like a sophisticated library organization system, where each book represents a piece of information. The librarian (the model) knows exactly which books to keep close for reference (important information to remember) and which ones can be put back on the shelf (less relevant information), ensuring that when someone asks for a specific detail, they can quickly find the right book, even if it was requested some time ago.

• Introduced in the paper 'Attention is All You Need'. • Replaces recurrence with self-attention mechanism. • Key Components: Attention, Multi-head attention, Positional encoding.

Transformers are a revolutionary architecture introduced to handle sequences more efficiently. Unlike RNNs and LSTMs, which process information sequentially, transformers use a self-attention mechanism that allows them to weigh the importance of different parts of the input data simultaneously. This means that any word can directly attend to all other words in a sequence without waiting for the preceding words to be processed first. Key components of transformers include attention mechanisms, which help in deciding how much focus to give to different parts of the input, multi-head attention, allowing the model to attend to different parts of the sequence in various ways, and positional encoding, which helps maintain the order of the words.

Imagine reading a book and being able to instantly look up references in different chapters at the same time. Instead of going page by page, you have an advanced search function that highlights connections throughout the entire book. This is akin to how transformers work; they can identify important relationships in data, making them exceptionally powerful for tasks like translation and text generation.

--