7.1.8. Cache Misses in Problem 2

Multi-level caches; now, with respect to single level caches, we also have multiple cache hierarchies. The primary cache or level one cache in multi-level caches is attached to the processor; it is small but fast. Added to that, we have a level 2 cache which services misses from the primary cache, it is typically larger in size, but slower than the primary cache; however, it is much faster than the main memory.

Multi-level caches are designed to enhance data retrieval speeds for CPUs. The primary cache (L1) is tiny but exceedingly fast, directly linked to the processor. When data isn't found in L1 (a cache miss), the L2 cache is consulted. L2 is larger but slightly slower than L1. However, both caches are still significantly quicker than accessing the main memory, allowing for efficient data retrieval and reduced processing times overall.

Think of caches as layers of a library. The L1 cache is like a small, well-organized library section where you can quickly grab books (data). If you can't find a book there, you check a bigger section in the library (L2 cache), which is less focused but has more titles. If the book isn't there either, you go to a massive warehouse (main memory) where the retrieval takes much longer.

Let us consider a CPU with a base CPI of one when all references hit the primary cache. Miss rate per instruction is 2 percent, meaning 2 percent of all the instructions miss the primary cache. The main memory access time is 100 nanoseconds.

In this scenario, if every instruction hits the primary cache, the cycles per instruction (CPI) is one. However, with a 2% miss rate, when data is not found in L1 and has to be fetched from main memory, the processor experiences a delay, significantly increasing the CPI. This example shows how cache misses directly impact performance by incurring substantial delays.

Imagine you're trying to find a book to complete your homework. If it's in your room (L1), you grab it quickly. If it's not, you might need to go to another room (main memory) to find it, taking a lot more time. If you frequently need to go to that other room, your homework will take much longer to finish.

The effective CPI will be 1 when I have a cache hit plus the penalty for misses. So, the effective CPI becomes 1 + 0.02 × 400 = 9 cycles.

When accounting for cache misses, the effective CPI is calculated by considering the original CPI (which is 1) and adding the contribution from the cache misses. Given a miss penalty of 400 cycles for a 2% miss rate, the total effective CPI rises to 9 cycles, indicating a significant impact on performance and efficiency.

This can be compared to a student who takes 1 hour to complete their homework if they can find all materials readily available. However, if every 20 minutes they spend an extra 40 minutes hunting for missing materials, their total homework time soars, making it seem like it takes nine times longer to complete.

Now let us assume that along with the primary cache, we have added an L2 cache. The L2 cache has an access time of 5 nanoseconds, and the global miss rate to main memory is 0.5 percent.

The addition of an L2 cache helps decrease miss penalties since it serves as an intermediary between the L1 cache and the main memory. The combined miss rate becomes lower due to fewer accesses to the main memory, allowing the CPU to execute instructions more efficiently.

Returning to the library analogy, if you have a helper (L2 cache) who can quickly find and fetch books from a storage room (main memory), your overall search time for books drastically reduces.

The effective CPI will be 1 plus the contributions from both levels of cache, resulting in a new effective CPI of 3.4, thus showing that adding the L2 cache made the processor 2.6 times faster.

With the introduction of the L2 cache, the frequency of accessing the main memory decreases, leading to a reduced effective CPI of 3.4. This demonstrates the significant performance enhancement achieved through multi-level caching systems.

If a student organizes their materials systematically and has a designated helper to fetch additional materials, they will significantly speed up their study sessions. Their efficiency increases; the same applies in computing with an efficient cache setup.

The focus of the primary cache is to minimize the hit time, while the focus of the L2 cache is on low miss rate so that memory access can be avoided.

The design of multi-level caches can be optimized to ensure quick data access (through quicker hit times in L1) and minimized delays when accessing L2. This allows the CPU to execute instructions faster and more efficiently due to reduced reliance on slower memory.

It's like a restaurant that efficiently takes orders from patrons (hit time) while ensuring that they have a waiting area (L2 cache) for when the kitchen is busy, thus reducing overall wait times for meals.

--