2. Word Embeddings and Representations

Static Embeddings: ● word2vec: Skip-gram and CBOW ● GloVe: Global Vectors for word co-occurrence

Static embeddings are fixed representations of words in a vector space. They do not change according to context. The 'word2vec' model creates these embeddings using two primary techniques:

1. Skip-gram: This approach predicts the surrounding words given a target word. For example, if the target word is 'bank', skip-gram would learn the words that frequently appear with it, like 'river' or 'money'.
2. CBOW (Continuous Bag of Words): This method does the opposite; it predicts the target word from the context words.

Another popular static embedding is GloVe, which stands for 'Global Vectors for word co-occurrence.' GloVe captures the relationship between words by analyzing the frequency with which they appear together in a large corpus. This helps create a vector representation that reflects the words' meanings based on their co-occurrence patterns.

Think of static embeddings like a dictionary entry for a word. Just as a dictionary provides a single definition for a word, static embeddings assign a fixed vector that captures the word's meaning without considering the sentence it's used in. For instance, the word 'bank' will have the same vector representation whether it's used in 'the bank of the river' or 'the bank where I deposit money.'

Contextual Embeddings: ● ELMo: Varies representations depending on context ● BERT/GPT: Deep transformer-based contextual understanding

Contextual embeddings represent words dynamically, depending on the specific sentence or context in which they appear. This is crucial because many words have multiple meanings.

1. ELMo (Embeddings from Language Models): ELMo generates embeddings by using a deep learning model that takes the entire sentence into account. This means the representation of the word 'bank' will change based on what other words are in the sentence. For example, in 'the bank of the river,' the embedding will reflect the meaning related to a geographic feature. In contrast, in the phrase 'I went to the bank,' it will represent the financial institution.

2. BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer): These models use advanced transformer architectures to achieve a deep understanding of context. BERT examines entire sentences bidirectionally, capturing the context from both previous and following words, while GPT generates text in a unidirectional manner, predicting the next word in a sequence based on the preceding words.

Think of contextual embeddings like an actor portraying a character in a movie. The actor's performance changes depending on the script, setting, and other characters' actions. Similarly, a word's meaning shifts based on its context in a sentence. For instance, the vibe of 'bank' shifts dramatically when paired with 'money' versus 'river,' much like how an actor's portrayal of a villain might change with different scenes and other characters.

Key Insight: "Bank" means different things in “river bank” vs. “savings bank”

This key insight emphasizes the importance of context in understanding language. The word 'bank' serves as an excellent example of a homonym, a word that has multiple meanings based on its usage. Without recognizing context, we could easily misinterpret the message. In language processing, distinguishing meanings based on surrounding words is critical for effective communication and machine understanding.

Consider this: if you heard someone say, 'I'm going to the bank,' you might imagine a financial institution. However, if they say, 'I'm sitting on the bank,' you'd picture a place beside a river. Just as a listener uses context clues to understand the correct interpretation, language models must decipher meaning in similar ways to process and respond accurately.

--