4.5. Cache Memory

Cache memory is based on the SRAM memory technology. It’s a small amount of fast memory which sits between the main memory and the CPU and it may be located within the CPU chip or a separate module plugged in on the motherboard.

Cache memory is a very fast type of memory that is located close to the CPU. Its main purpose is to speed up the process of accessing data that the CPU frequently uses. By keeping this fast memory close by, the CPU can retrieve data much quicker than if it had to fetch it from the slower main memory every time. Cache can be found inside the CPU itself or as a separate module on the motherboard.

Think of cache memory like a chef's workspace in a busy kitchen. Instead of going all the way to the pantry (main memory) each time a chef needs an ingredient, they keep a small selection of frequently used items on their counter (cache memory). This way, they can grab what they need quickly and keep cooking without wasting time.

When the processor attempts to read a memory word from the main memory, it places the address of the memory word on the address bus. A check is made to determine if the word is in cache. If the word is in cache we have a cache hit; otherwise, we suffer a cache miss.

When the CPU tries to read data, it first looks in the cache to see if it is there. If the data is found, this is called a 'cache hit'. If not, it's called a 'cache miss', and the CPU has to fetch the data from the slower main memory. This process of checking the cache first is essential for speeding up computations, as reducing the number of cache misses can greatly improve performance.

Imagine you're at a library. When searching for a book, you first check the shelf where the most popular and frequently borrowed books are kept (cache). If the book is there (cache hit), you take it out immediately. But if it’s not there (cache miss), you have to go to the larger storage room (main memory), which takes more time.

The hit time is the time to access a memory word in case of a hit. The fraction of memory accesses resulting in hits is called the hit ratio or hit rate and is defined as the number of cache hits over a specific number of accesses. Conversely, the miss ratio is 1 minus the hit ratio.

The hit ratio gives us a measure of how effective the cache memory is—how often the data the CPU needs is found in the cache. The miss ratio, on the other hand, tells us how often the CPU has to go to the slower main memory for data. A higher hit ratio indicates better cache performance, meaning less waiting time for the CPU.

Continuing with the library analogy, if you checked the popular shelf and found your book 70 times out of 100 visits, your hit ratio is 70%. Your miss ratio would be 30% because you had to search elsewhere for the book 30 times. A library that organizes its bestsellers effectively increases its hit ratio, just like a good cache does for CPU data.

In case of a cache miss, a block of memory consisting of a fixed number of words is read into the cache, and then the word is delivered to the processor. A block of memory is fetched instead of only the requested memory word to take advantage of the locality of reference.

When a cache miss occurs, rather than just getting the single piece of data requested, the system fetches a whole block of data from memory. This is because of the principle of locality of reference: programs tend to access data in clusters. By fetching more data at once, the chances of hitting the required data on subsequent accesses increase, thereby improving efficiency.

Imagine you're packing a suitcase for a trip. You wouldn’t just pack one shirt if you know you’ll need more clothes; you'd pack a few items at a time (block fetch). This way, you're prepared for several outfits without needing to repack constantly. Similarly, the cache fetches a block of data to better meet future needs.

Here in the figure, we see that CPU asks for a word from memory and if that word is present in cache, we send it back to the CPU and this is word transfer. If this word is not present in cache, we have a cache miss and then we fetch a block from main memory.

This part of cache architecture illustrates the fundamental process of data retrieval. When the CPU makes a request for data, it first checks the cache. If the data is there, it performs a fast 'word transfer' directly. If it’s not available and a 'cache miss' occurs, the system retrieves a whole block of data from the slower main memory, illustrating the performance trade-offs made in memory architecture.

Think of this process as a waiter in a restaurant. If a customer orders a meal and it's available in the kitchen (cache), the waiter brings it out quickly. If the dish isn't readily available and needs to be prepared (cache miss), the waiter must go back to the kitchen, potentially taking longer to serve the customer.

We may have different levels of cache not only a single level of cache. We have CPU followed by a small very fast cache, followed by a level 2 cache, which is slower than level one cache but is also higher in capacity.

In modern systems, there are several levels of cache (L1, L2, and sometimes L3) each serving different speeds and sizes. L1 cache is the fastest and smallest, directly within the CPU. L2 is slower and larger, and sometimes L3 exists further away from the CPU but has even more capacity. This hierarchical structure helps balance the trade-offs between speed, size, and cost.

Think of levels of cache like layers of a multi-story parking garage. The top floor closest to the entrance (L1) allows the fastest access, but has the least space. The middle floors (L2) offer more parking but take a little longer to reach, while the bottom floor (L3) has the most space but takes the longest walk from the entrance. This setup allows for efficient vehicle access based on different needs.

--