21.7. Expected Length of the Encoding

So, this brings us to the idea having a variable length encoding, where we use different strings of different length for different letters in the alphabet. One of the most famous examples of the variable length encoding is the classical Morse code, which is developed by Samuel Morse from the telegraph who is invented.

Variable length encoding refers to the method of assigning different lengths of code to different characters, depending on their frequency of use. This means more common characters can be represented with shorter codes, while less common characters use longer codes. The example of Morse code showcases this concept, where frequently used letters like 'e' are shorter than others, allowing for a more efficient communication method.

Imagine you are packing a suitcase. You have shirts (common items) that you can fold up tightly, taking less space, while bulky jackets (less common items) require more room. Similarly, variable length encoding optimizes the space needed for communication by assigning shorter codes to 'packed' frequently used letters.

In order to make a variable length code an unambiguous decodable, we need what is called a prefix quantity. When we read through a sequence of 0’s and 1’s, we should be ambiguously clear, whether we have read a letter or there is more to read.

A prefix code ensures that no code is a prefix of another, which allows for unambiguous decoding. This means if you encounter a sequence of bits, you can definitively determine the end of one character and the start of another. If the coding were ambiguous, it would lead to confusion, as different interpretations could occur based on how you read the binary sequence.

Consider a set of traffic signals. If the red and green lights were indistinguishable, drivers wouldn't know when to stop or go, leading to confusion and chaos. Prefix coding ensures clarity in communication, similar to clear traffic signals guiding drivers correctly.

Now, we have a message, it consists of some n symbols. So, we have M 1, M 2 up to M n, so these are n symbols. Now, we know that if I take a particular letter x, then f x fraction of these are x, so in other words, if I take n and I multiply by a fraction is say, if I fix is say one third, then one third of n.

The expected length of encoding is calculated based on the frequency of each symbol in the message and the length of each encoding. For example, if 'x' appears frequently in a text, it will have a shorter encoding. By multiplying the frequency (f) of 'x' by the length of its encoding, and summing this for all symbols, you can determine the total number of bits required to encode the message.

Think of a grocery store checkout where each item has a price tag. If you buy mostly inexpensive items, your total cost (encoding) would be lower than if you were mostly purchasing expensive items. The frequency and the price of items give you a clear picture of what you're spending, similar to how frequency and encoding length help calculate the total bit length needed.

So, now, our goal is to find an assignment capital E which minimizes this quantity. So, in our coding the average efficient is possible.

To achieve an optimal encoding, we want to assign shorter codes to more frequently occurring letters. This involves analyzing the frequencies of letters in a given dataset and devising a coding scheme that reflects these frequencies, reducing the overall average length of the encoded message.

Imagine a library where popular books are placed in the front for easy access while rare books are on higher shelves. This organization similar to encoding ensures that the most accessed resources (letters) are quicker to get to (shorter codes), while the infrequent ones, though accessible, take more time (longer codes) to reach.

--