2.6.1. Cache Hit and Miss

Cache memory as we said is based on the SRAM memory technology. It is a small amount of fast memory which sits between the main memory and the CPU and it may be located within the CPU chip or as separate modules plugged into the motherboard.

Cache memory acts as a buffer between the CPU and main memory. It uses SRAM technology which allows it to be significantly faster than DRAM used in main memory. The small size and high speed of cache means that it can quickly provide data that the CPU needs, improving overall performance.

Think of cache memory like a chef's prep station in a restaurant. Just as a chef keeps frequently used ingredients close by for quick access while cooking, a CPU keeps frequently accessed data in cache memory to speed up processing.

When the processor attempts to read a memory word from the main memory, it places the address of the memory word on the address bus. A check is made to determine if the word is in cache. If the word is in cache we have a cache hit; otherwise, we suffer a cache miss.

A cache hit occurs when the data requested by the CPU is found in the cache memory. This allows for rapid data retrieval, hence speeding up performance. A cache miss happens when the data is not in the cache, requiring the system to fetch it from the slower main memory. This process naturally takes more time, slowing down the overall operation.

Imagine playing a trivia game where you memorize answers to common questions (cache). When a question is asked, if you remember the answer, you answer quickly (cache hit). If not, you have to look it up in a textbook (cache miss), which takes longer.

The time to access a memory word in the case of a hit is the hit time. The fraction of memory accesses resulting in hits is called the hit ratio or hit rate and is defined as number of cache hits over a certain given number of accesses to the memory.

Hit time is critical in determining how quickly a system can respond to the CPU's requests. The hit ratio helps evaluate the effectiveness of the cache; the higher the ratio, the more efficient the cache is at serving the data requests without resorting to slower memory accesses.

Consider a library that has a section reserved specifically for frequently borrowed books (cache). If the librarian can quickly provide a popular book from that section (high hit rate), it saves patrons time compared to searching the entire library (low hit rate).

In case of a cache miss, a block of memory consisting of a fixed number of words is read into the cache and then the word is delivered to the processor. The time to replace a cache block and deliver the requested word to the processor is known as miss penalty.

When a cache miss occurs, not only the requested word needs to be fetched, but also a whole block of words is brought into the cache. This strategy capitalizes on the principle of locality of reference, as future memory requests are likely to access data within this block. The miss penalty reflects the increased time it takes to retrieve these data from the slower main memory.

Returning to the library analogy, if you request a book that isn’t on the popular shelf, the librarian must locate the whole collection related to that book topic (miss penalty), which takes longer than getting a book that’s already within reach.

If the word is present in cache, we send it back to the CPU and this is word transfer. If this word is not present in cache, we have a cache miss and then we fetch a block from main memory and this block contains the desired word as well.

The process of transferring data varies significantly depending on whether a cache hit or miss occurs. A hit involves a simple transfer of the requested word back to the CPU, while a miss requires fetching an entire block from memory, subsequently more complicated and time-consuming.

Picture a takeout restaurant. If your favorite meal is ready (cache hit), it’s handed to you directly. If you need something not on the menu (cache miss), the staff must prepare a whole new order from scratch, which takes longer.

The figure on the right shows that we may have different levels of cache not only a single level of cache.

Modern computer architectures often implement multiple levels of cache (L1, L2, L3), where L1 is the smallest and fastest, followed by larger and slower L2 and L3 caches. This hierarchy allows a more efficient structure for data retrieval, optimizing performance based on access frequency.

Think of a multi-tiered shopping center, where the high-end boutique (L1 cache) has the latest fashion that sells quickly, the mid-range store (L2) has more variety, and the large warehouse (L3) holds a lot, but takes longer to browse through.

--