7.1.3. Effective CPI with L2 Cache

Multi-level caches; now, with respect to single level caches, we also have said before that we have multiple cache hierarchies. Now we will talk about them here. The primary cache or level one cache in multi-level caches is attached to the processor; it is small bus but fast. Added to that we have a level 2 cache which services misses from the primary cache, it is typically larger in size, but also slower, but slower than the primary cache; however, being much faster than the main memory.

In computing, caches are small, fast storage locations that hold copies of frequently accessed data from the main memory. A multi-level cache system typically has a Level 1 (L1) cache, which is small but extremely fast, located closest to the CPU. The Level 2 (L2) cache is larger and somewhat slower than the L1 cache but is still much faster than accessing main memory. This hierarchy helps to speed up data access times by storing copies of frequently accessed data closer to the CPU. Therefore, when the CPU requires data, it first checks the L1 cache, then the L2 cache before resorting to the slower main memory.

Think of a multi-level cache as a library system. The L1 cache is like having books on your desk that you use every day—easy and quick to access. The L2 cache is akin to having a closet in your room filled with other books that you use frequently but not daily, making it a bit slower to access because you have to get up. Finally, the main library represents the main memory, which is larger and contains all the books but takes much longer to reach.

So, what hierarchy do I have? I have the processor, then from the processor I have a small primary cache, typically these have these are separate data and instruction caches, then a basically I have a combined much bigger in size L2 cache, this will be these are L1 these both are L1 the L2 cache, and this is then attached to the main memory; which is much bigger.

In a multi-level cache system, efficiency is essential. Given that there are separate cache regions for data and instructions within the primary cache (L1), any missed data requests lead to looking up the larger L2 cache, which services the primary cache misses. If data isn't found in either cache, the system must access the significantly slower main memory. The efficiency of the cache is measured by how often data can be retrieved from L1 or L2 caches without needing to access the slower main memory. The calculation of effective cycles per instruction (CPI) considers both cache hits and misses.

Imagine you're baking in the kitchen. If your commonly used ingredients (like flour and sugar) are readily available on the countertop (L1 cache), it’s quick to grab them. If an ingredient is in the pantry (L2 cache), it takes a little longer to fetch. However, if you're out of everything and have to go to the store (main memory), it takes the longest time. The goal when cooking, much like when processing data, is to minimize those trips to the store.

Now, let us assume that with this along with this cache we have added a L2 cache ok. The L2 cache has an access time of 5 nanoseconds ok. To get to the L2 cache I require to get to the L2 cache and get the data I require 5 nanoseconds. Now, the global miss rate to main memory is 0.5 percent. So, the number of the percentage of cases for which I miss the primary cache as well as the secondary cache means the L2 cache is only 0.5 percent.

By adding an L2 cache, which has a faster access time than the main memory (5 nanoseconds compared to 100 nanoseconds), the overall efficiency of the CPU increases immensely. With a lower miss rate of 0.5% for the L2 cache after an L1 cache miss, the system becomes better at fetching data. This decrease in miss rates translates to fewer penalties when the CPU needs to access data, leading to a reduced effective CPI.

Analogy of a customer service line can help illustrate this: if the main customer service counter (main memory) is slow, a fast, secondary service desk (L2 cache) that can handle quick queries improves overall customer satisfaction. The small number of customers needing to access the main counter (0.5% misses in cache) ensures that most customers get their questions resolved quickly at the secondary desk.

So, the total effective CPI becomes 1 plus 2 percent of the times I missed the primary and go to the L2, and 0.005 times I miss the L2 and go to the main memory. So, the effective CPI will be 3.4. Now, the performance ratio without and with the secondary cache will therefore be 2.6.

The calculation of effective CPI incorporates not only the standard CPI when fetching data from the L1 cache but also factors in the penalties incurred during cache misses. By factoring in the low miss rates from both caches, the overall effective CPI is a crucial measure that informs the performance of the CPU. When L1 and L2 caches work efficiently together, the effective CPI decreases significantly, demonstrating increased processing speed.

Consider a delivery service. If you can rely on a local consolidator (L1 cache) for quick deliveries, it saves time. However, if a few items are missing, you can get them from the nearby warehouse (L2 cache), which is still quicker than driving all the way back to the original warehouse (main memory). A few times needing to drive to the warehouse impacts your overall delivery time, which is like the effective CPI reducing when caches are optimized.

Now, multi-level cache design issues. So, the focus of the primary cache is to maximize hit time. Because I expect that each time I go for executing an instruction I will go to the cache and I have to fetch. So, I will try to minimize the amount of time I require to access the primary cache and get data.

When designing multi-level caches, priorities shift based on the cache level. The primary cache is designed with minimal hit time in mind, ensuring that data can be retrieved as quickly as possible for instruction execution. In contrast, the L2 cache focuses on minimizing misses, as it comes into play only when the primary cache fails to provide requested data. This balance in cache sizes and access times is critical for maintaining efficient CPU performance.

Consider a restaurant: the chef (CPU) wants to quickly access the ingredients (data). The ingredients are kept on the cooking counter (L1 cache) for quick access, while bulk supplies are stored in a nearby pantry (L2 cache). The goal is for the chef to limit trips to far-off storage (main memory), meaning your customers get served tastier meals faster.

--