7.1. Multi-level Caches

Multi-level caches; now, with respect to single level caches, we also we have said before that we have multiple cache hierarchies. Now we will talk about them here. The primary cache or level one cache in multi-level caches is attached to the processor; it is small but fast. Added to that we have a level 2 cache which services misses from the primary cache, it is typically larger in size, but also slower, but slower than the primary cache; however, being much faster than the main memory. So, the main memory then services L2 cache.

Multi-level caches are designed to enhance processing speed and efficiency compared to single-level caches by utilizing multiple layers or levels of cache memory. The Level 1 (L1) cache is directly attached to the processor, which allows for very fast access times. However, it is small in size to facilitate quick access. The Level 2 (L2) cache comes in to help when there is a cache miss in the L1 cache, meaning that if the required data is not found in the L1 cache, the processor can check the larger, but slower, L2 cache before going to the main memory. The main memory, while being the largest in size, has the slowest access time among all these caches.

Think of the multi-level caches like a library system. The L1 cache can be seen as a personal bookshelf in your room where you keep your most frequently read books (quick access but limited space). The L2 cache is like a larger bookshelf in a neighboring room, where you have more books that you occasionally refer to (larger but takes slightly longer to access). The main memory is like the library across town, which has an extensive collection of books, but traveling there takes a lot more time.

So, what hierarchy do I have? I have the processor, then from the processor I have a small primary cache, typically these have these are separate data and instruction caches, then a basically I have a combined much bigger in size L2 cache, this will be these are L1 these both are L1 the L2 cache, and this is then attached to the main memory; which is much bigger. Now, in more in a more high end machines we also have 3 levels of cache. Typically, L1 or L2 are on chip, sometimes L3 cache is off chip in typically L3 caches are off chip.

In the cache hierarchy, the processor interacts primarily with the L1 cache first. This cache is divided typically into two parts: one for data and one for instructions. Beyond the L1 cache is the L2 cache, which is larger and designed to capture misses from the primary caches. In modern high-performance systems, there's even a Level 3 (L3) cache, which is usually larger again and often located off the processor chip. The goal is to create layers of caches that balance size and access speed, ensuring that the most commonly accessed data is as close to the processor as possible.

Imagine a fast food drive-thru system. The point of order (L1 cache) provides quick service with a limited menu (fast but small). If you need something not available at the drive-thru (a cache miss), you go to a larger but slightly slower service window (L2 cache). If your request isn't fulfilled there either, you might have to go into the full restaurant (main memory), which has a larger menu but takes more time, reflecting the increasing distance and time cost associated with larger caches.

Now, we will take an example on the use of multi-level caches. Particularly, we will see how multi-level caches are able to reduce miss penalties. Let us consider a CPU with a base CPI of one when all references hit the primary cache. So, cycles per instruction is one cycle cycles per instruction is 1. So, one cycle instruction execution when all references hit the primary cache, the clock rate is 4 gigahertz. Miss rate per instruction is 2 percent.

In this section, we analyze how multi-level caches impact performance, specifically through miss penalties. We begin with a base Cycles Per Instruction (CPI) of one, indicating that ideally, each instruction takes one cycle to execute. However, with a miss rate of 2%, which means one out of fifty instructions misses in the primary cache, there will be a performance penalty as the CPU must then access the slower main memory to retrieve the necessary data. This relationship between the clock rate, miss rates, and effective CPI illustrates the importance of cache layers in improving execution efficiency.

Continuing the restaurant analogy, consider that each customer ideally gets served immediately (base CPI). If a customer orders something that isn't available at the drive-thru (primary cache miss), they will have to wait longer while the staff goes to the kitchen to fetch that item (main memory access), thus increasing the overall wait time for service. This increase in wait time can be visualized as the miss penalty impacting the efficiency of the entire process.

Now let us assume that with this along with this cache we have added a L2 cache ok. The L2 cache has an access time of 5 nanoseconds ok. To get to the L2 cache I require to get to the L2 cache and get the data I require 5 nanoseconds. Now, the global miss rate to main memory is 0.5 percent.

Assuming we've added a Level 2 cache, which has a significantly faster access time of 5 nanoseconds, we must consider the overall effective CPI now. With this new architecture, the miss rate for the combined L1 and L2 caches has reduced to 0.5%, meaning that now only one out of every two hundred instructions results in another miss that forces us to access the main memory. These adjustments lead to a calculation of the Effective CPI that incorporates both cache levels, resulting in better overall performance.

If we think of our restaurant system, adding the L2 cache is like introducing a dedicated kitchen staff only to prepare faster, commonly ordered items. Customers experience less waiting when they require something, as fewer items need to be fetched from the main store (main memory), reducing overall service time significantly, akin to lowering the Effective CPI.

Now, multi-level cache design issues. So, the focus of the primary cache is to minimize the hit time. Because I expect that each time I go for executing an instruction I will go to the cache and I have to fetch. So, I will try to minimize the amount of time I require to access the primary cache and get data.

Multi-level cache design requires careful consideration. The primary cache is designed primarily for speed with a focus on minimizing hit time, meaning the average time it takes to successfully retrieve data from the cache. In contrast, the L2 cache is aimed at minimizing miss rates to reduce the frequency of accessing the significantly slower main memory. The L1 cache needs to be smaller than a theoretical optimal single-level cache to achieve these speed gains, while the L2 cache can afford to be larger, enhancing its overall efficiency.

Imagine your personal bookstore (L1 cache) has a small selection of books (a focus on speed), while another library (L2 cache) has a broader selection of books but is somewhat slower to access. You choose to small bookshelf for instant access, knowing that if you can’t find what you need there, you can go to the bigger library with a hope of finding your book faster than traveling clear across town to a mega bookstore (main memory).

--