An Engineer's note on sliding window attention

Category: AI Trends

By Isham Rashik

In traditional transformer models, self-attention lets every token attend to every other token in a sequence. Thatโ€™s powerful, but it comes with a quadratic compute and memory cost as sequence length grows making long context processing expensive and slow. Sliding Window Attention solves this ...

๐Ÿ”Ž ๐’๐ฅ๐ข๐๐ข๐ง๐  ๐–๐ข๐ง๐๐จ๐ฐ ๐€๐ญ๐ญ๐ž๐ง๐ญ๐ข๐จ๐ง โ€” ๐–๐ก๐š๐ญ ๐ˆ๐ญ ๐ˆ๐ฌ & ๐–๐ก๐ฒ ๐ˆ๐ญ ๐Œ๐š๐ญ๐ญ๐ž๐ซ๐ฌ ๐ข๐ง ๐‹๐‹๐Œ๐ฌ


In traditional transformer models, self-attention lets every token attend to every other token in a sequence. Thatโ€™s powerful, but it comes with a quadratic compute and memory cost as sequence length grows โ€” making long context processing expensive and slow.

Sliding Window Attention solves this by having each token attend only to a fixed local window of neighboring tokens rather than the whole history. This drops complexity from O(Lยฒ) to roughly O(L ร— W), where W is the window size โ€” a huge win for efficiency without tanking performance on many tasks.


๐Ÿ”ฅ ๐–๐ก๐ž๐ซ๐ž ๐˜๐จ๐ฎ ๐’๐ž๐ž ๐“๐ก๐ข๐ฌ ๐ข๐ง ๐ญ๐ก๐ž ๐–๐ข๐ฅ๐: ๐Œ๐ข๐ฌ๐ญ๐ซ๐š๐ฅ & ๐†๐ž๐ฆ๐ฆ๐š 3


๐Ÿง  ๐Œ๐ข๐ฌ๐ญ๐ซ๐š๐ฅ

Mistral models adopt sliding window attention to limit the token horizon each layer attends to, drastically reducing memory and compute during inference. Rather than every token attending to all past tokens, it attends to a fixed set of recent ones โ€” giving the model practical long-context handling with much lighter resource use.


This doesnโ€™t mean Mistral โ€œforgetsโ€ the rest of the context โ€” information still percolates across layers โ€” but it focuses compute where itโ€™s most cost-effective.


๐Ÿš€ ๐†๐ž๐ฆ๐ฆ๐š 3

Gemma 3 takes this further with a hybrid scheme:

- Most layers use sliding window (local) attention with a smaller window (e.g., ~1 k tokens), focusing on immediate context.

- Periodically, a global attention layer attends to the full history, keeping long-range dependencies intact.

- The mix (e.g., 5 local layers for every global layer) favors efficiency with minimal performance impact.


Compared to its predecessor, Gemma 3 shrinks the window size and leans harder on local attention โ€” a deliberate trade-off to cut memory costs while still supporting huge context lengths.


๐Ÿ’ก ๐–๐ก๐ฒ ๐˜๐จ๐ฎ ๐’๐ก๐จ๐ฎ๐ฅ๐ ๐‚๐š๐ซ๐ž

Sliding window attention is a practical engineering strategy โ€” not a magic bullet. Itโ€™s about ๐ฌ๐œ๐š๐ฅ๐ข๐ง๐  ๐œ๐จ๐ง๐ญ๐ž๐ฑ๐ญ ๐œ๐ก๐ž๐š๐ฉ๐ฅ๐ฒ:

โœ… Reduces attention compute and KV cache memory

โœ… Enables processing of very long texts efficiently

โš ๏ธ Slightly limits direct global token interactions (mitigated with occasional global layers)


For real-world use cases โ€” long document summarization, codebases, legal text โ€” this trade-off often pays off big.


๐Ÿ“Œ Bottom Line


Modern LLMs like Mistral and Gemma 3 are embracing sliding window attention because itโ€™s one of the most effective tricks today to push context scaling without exploding costs. Itโ€™s solid engineering: simple, efficient, and now battle-tested in cutting-edge architectures.

Read the full article on Garage Labs Tech โ€” India's applied AI education platform. Explore our AI courses and programmes.