Impact of Cache Memory on System Performance

Impact of Cache Memory on System Performance

1. What a Cache Is

A cache is a small, very fast SRAM memory placed between the CPU and the much slower DRAM main memory. It keeps a copy of recently used blocks of memory so that most CPU requests are served at SRAM speed instead of DRAM speed. It is a hardware device — software neither addresses it nor manages it, it simply observes that the same memory addresses keep being requested.

2. Why It Works — Locality of Reference

Cache performance is a direct consequence of program behaviour:

• Temporal locality — the same instruction/data is reused shortly (loop counter i, repeatedly called function) ⇒ keep the block, it will be asked for again.
• Spatial locality — nearby addresses are used next (array traversal, sequential instructions) ⇒ fetch a whole block/line (typically 64 B) rather than a single word.

If access were uniform and random, a cache would be useless.

3. Key Parameters

• Hit — requested block found in cache; Miss — must fetch from lower level.
• Hit ratio $h$ (typical 90-99% for L1 data caches) and miss ratio $(1 - h)$.
• Hit time — cache access time; miss penalty — extra time to fetch the block from the next level.
• Block/line size — 32-64 B in practice. Very small blocks lose spatial locality; very large blocks evict useful data and increase the miss penalty (the classic "miss rate vs block size" curve falls, then rises).
• Associativity — direct-mapped (fastest, most conflict misses), fully associative (fewest misses, costliest), k-way set associative (e.g. 8-way: the practical compromise).
• Replacement policy — LRU (best but costly), FIFO, random, LFU.
• Write policy — write-through updates cache and memory together (simple, more traffic); write-back updates only the cache and writes memory when a dirty block is evicted (less traffic, more complexity).

4. Impact on Performance — Numbers

Average memory access time (AMAT) with one cache level:

Numerical for access time: $T_1 = 2$ ns (cache), $T_2 = 60$ ns (DRAM), $h = 0.95$:
$T_{ave} = 0.95 × 2 + 0.05 × 60 = 1.9 + 3.0 =$ 4.9 ns versus 60 ns without cache → 12× faster average access.

Numerical for CPI / speed-up: suppose a memory access costs 1 cycle in cache and 50 cycles in DRAM, and $h = 0.95$:

Without any cache every access costs 50 cycles → speed-up $= 50 / 3.45 ≈$ 14.5× — the machine executes the same program roughly 14 times faster purely because of the cache. Note how sensitive this is: dropping $h$ from 0.95 to 0.90 gives $CPI = 0.9 + 5 = 5.9$ cycles, i.e. a 41% loss in speed for a 5% drop in hit ratio.

5. Multi-Level Caches and Other Aids to Performance

• L1 — split into I-cache (instructions) and D-cache (data) so that fetch and decode can proceed in parallel; 32-64 KB per core.
• L2 — larger, unified, on-chip; catches L1 misses.
• L3 — shared by all cores, several MB; catches L2 misses and avoids DRAM traffic.
• Adding a level helps only when its hit time is much smaller than the miss penalty it saves (inclusion property: L1 ⊂ L2 ⊂ L3 ⊂ RAM).
• TLB (Translation Lookaside Buffer) — a cache of page-table entries, removing the cost of virtual-to-physical address translation.
• Prefetching and write buffers — fetch the next block speculatively / let the CPU continue while a write completes.
• Amdahl's law bounds the gain: if a fraction $p$ of execution is sped up $k$ times, overall speed-up is

so improving only memory (p = 0.5, k = 10) gives $S = 1/(0.5 + 0.05) = 1.82×$, not 10×.

6. Practical Summary

Cache converts a memory system that behaves like DRAM (60-100 ns) into one that behaves almost like SRAM (2-5 ns) for typical programs. The cost is a small area of fast SRAM, complex control logic and the need for coherence protocols in multicore systems (MESI etc.).

7. Exam Points

• What is cache memory? Explain how it improves performance, using locality of reference — 5-7 marks, very frequent.
• Derive/apply $T_{ave} = h × T_1 + (1 - h) × T_2$; numerical — 3-5 marks.
• Explain write-through vs write-back and LRU replacement — 5 marks.
• Difference between L1, L2 and L3 cache — 3-5 marks.
• Differentiate direct-mapped, fully associative and set-associative mapping — 5 marks.
• Numerical combining hit ratio, memory cycles and CPI to find speed-up — 5 marks.