System Prompts & CLAUDE.md: The Foundation of Context Engineering

System Prompts & CLAUDE.md

A comprehensive guide to the foundational layer of context provided to large language models — what goes into a system prompt, how to structure it, and how to engineer it for maximum effectiveness.

This guide dives deep into system prompts: the foundational layer of context provided to large language models. In Claude Code, the primary vehicle for system prompts is the CLAUDE.md file.

It covers the five components of a well-structured system prompt (identity, rules, format, knowledge, tools), the "right altitude" principle for writing instructions, iterative construction methodology, selective retrieval of sections, the file hierarchy (CLAUDE.md, AGENTS.md, SKILL.md, SOUL.md), few-shot example strategies, and the coding agent landscape including Claude Code, Google Antigravity, Cursor, and Copilot.

Building on the six-layer context model from the companion guide on context window fundamentals, this guide focuses entirely on Layer 1: the system prompt.

Recap: The Six Layers of Context

The six elements that constitute the context provided to an LLM are:

1. System Prompt — rules, guardrails, identity (CLAUDE.md / AGENTS.md)
2. User Instructions — the direct prompt or query from the user
3. Tools — functions, APIs, libraries accessible to the model (MCP)
4. Memory / State — persistent knowledge (preferences, historical patterns)
5. RAG — dynamically fetched data from knowledge bases
6. Conversation History — the current session's message exchanges

These six elements are not listed in any particular priority order. The order in which they appear in the context window is what matters.

Memory vs. Conversation History

A common point of confusion: memory and conversation history are not the same thing.

Concrete example: An email agent responding to a thread. The 25 prior exchanges in that specific thread form the conversation history. How the user has replied to technical vs. non-technical emails over the past year, distilled from 5,000–10,000 email exchanges, forms the persistent memory/state.

Context Ordering and the Lost-in-the-Middle Effect

The context window has a well-documented attention bias. LLMs give the most prominence to the beginning and end of the context window. Content in the middle receives slightly less attention: this is the "lost in the middle" effect.

• System prompt (rules, guardrails) goes at the beginning: highest priority
• User prompt goes at the end: also high priority
• Everything else (RAG output, tool results, memory) sits in the middle

Even if an LLM supports 1–2 million tokens as its context window, try to keep your actual context length to 50,000–100,000 tokens. Larger contexts amplify the lost-in-the-middle effect.

Anatomy of a System Prompt (CLAUDE.md)

A CLAUDE.md file typically contains five components. These form the skeleton of any well-structured system prompt.

Identity

Identity is the first thing written in CLAUDE.md and often the least important in terms of behavioral impact. It describes what the project is — not in anthropomorphic terms ("you are a senior code reviewer"), but in functional terms.

• This is an email responding agent
• This is a personal digital clone for responding to Slack messages
• This is a website that automatically updates when new developments happen in the AI space

Identity does not mean "who the LLM is." It means what the thing you are building is meant to do. Avoid giving human-like personas unless specifically needed.

Token characteristics: Identity is typically one paragraph — it consumes the fewest tokens of all five components. If forced to cut sections, identity is the safest to remove first because the model can often infer its role from the rules, examples, and conversation context.

Rules

Rules define how the model should or should not behave. These are the guardrails — the component that most directly determines whether the system produces correct, safe, and consistent outputs.

• Never respond about refund policy without consulting the policy document
• Never use camelCase — use snake_case
• Never store student scores with more than two decimal places
• If a customer asks about returns, consult the return policy document before responding

Priority: Rules are the highest priority component. Even under extreme token constraints, rules should be the last component removed. Rules must be loaded into the context for every single API call because guardrails should never be broken.

If it says "never do X" or "always do Y": it is a rule. If it describes what exists or how things are structured: it is knowledge.

Format

Format specifies how the LLM should structure its output — independently of what it says or how it behaves. This includes output format (Python, JSON, CSV), project folder structure, and development commands.

src/
├── calculator.py # calculate_final_grade function
├── models.py # data classes
├── utils/
└── tests/

Knowledge

Knowledge describes what the reality of the application is: factual information the model needs to do its job — not instructions on how to behave, but the facts it must know to apply those instructions correctly.

• Environment variables and how they are structured
• Development phases (Phase 1: MVP, Phase 2: Multi-user)
• Tech stack details (Python, ChromaDB, LangChain)
• How the project overview describes the application

Tools

Tools define what the model can access: APIs, libraries, packages, and functions the model is permitted and expected to use.

• Use Claude API for LLM calls
• ChromaDB for vector storage
• LangChain or LlamaIndex for orchestration
• CSV.writer for export operations

Blurred Boundaries Between Components

The boundaries between components are fluid, not rigid:

Context engineering is not physics: it is more like a collection of thumb rules that evolved from experimentation by large companies and communities. The boundaries between knowledge, rules, and format are deliberately soft.

Right Altitude Principle

Every instruction in CLAUDE.md has a cost: it consumes tokens, adds noise, and competes with every other rule for the model's attention. Anthropic's documented guidance is direct — keep CLAUDE.md short and human-readable, and apply a single test to every line:

Would removing this line cause Claude to make mistakes? → Keep it. It encodes something the model cannot infer on its own.

Would removing this line make no difference? → Cut it. Bloated CLAUDE.md files cause Claude to ignore your actual instructions.

This maps onto two failure modes: instructions too vague to act on, and instructions so specific they break the moment the world changes.

Saying "do not hallucinate" is the most vague thing you can ever write in a system prompt. It provides zero actionable guidance to the model.

CLAUDE.md has a lifespan of weeks to months. Hardcoding specific prices, error messages, or API responses means the file becomes stale the moment any detail changes.

What to Include vs. Exclude

Include

• Bash commands Claude cannot guess
• Code style rules that differ from defaults
• Testing instructions and preferred test runners
• Repository etiquette: branch naming, PR conventions
• Architectural decisions specific to your project
• Developer environment quirks, required env vars
• Common gotchas and non-obvious behaviors

Exclude

• Anything Claude can figure out by reading code
• Standard language conventions Claude already knows
• Detailed API documentation (link to docs instead)
• Information that changes frequently
• Long explanations or tutorials
• File-by-file descriptions of the codebase
• Self-evident practices like "write clean code"

Review it when things go wrong, prune it regularly, and test changes by observing whether Claude's behaviour actually shifts. If Claude keeps violating a rule despite it being written down, the file is probably too long and the rule is getting lost in the noise. Add emphasis — IMPORTANT or YOU MUST — for rules that must not be ignored.

System Prompt Formatting: XML vs. Markdown

Use Markdown for CLAUDE.md. The difference in output quality between XML and Markdown is not perceivable in practice. Markdown is also the natural choice if you ever write system prompts for other platforms (Cursor, Copilot, Windsurf), since those tools use their own Markdown-based config files.

Markdown heading hierarchy enables section-based retrieval:

• # = top-level section
• ## = subsection
• ### = sub-subsection

Each heading creates a natural chunk boundary that can be independently retrieved. This property becomes critical as CLAUDE.md files grow larger — Markdown structure is what makes selective retrieval possible at all.

Iterative Construction: Start Minimal, Then Add

The temptation when writing CLAUDE.md is to define every rule upfront. Anthropic's recommendation is the opposite: start with the minimum viable system prompt and add rules only when observed failures demand them.

"You are a customer support agent for ShopCo." Run on real tasks. Failure: model hallucinates return policy.

"Use provided tools for actions. Do not guess." +Anti-hallucination. Failure: approves refund outside window.

"Returns must be within 30-day window." +Empathy-first, de-escalation. Failure: routes damaged item as standard return.

"Damaged items require different routing." +Edge cases, routing rules. Result: all tests pass within ~200 token budget.

Each failure reveals a missing rule or piece of knowledge. The scores progress from 3/10 to 9/10 over four rounds, with the total prompt growing from ~30 tokens to ~200 tokens. Every rule earns its place by fixing a real failure, not through speculation.

Avoiding Contradictions

The primary danger of writing a comprehensive system prompt in one shot is self-contradiction.

When defining 10–20 rules at once, some rules may conflict. The model then picks arbitrarily, and picks wrong.

Rule: Refund window must be within 30 days
User preference: If the case is legitimate, always provide the refund
Conflict: What if the case is legitimate but it is day 45? The model cannot satisfy both rules simultaneously.

The least you can do is either iterate and build, or build the CLAUDE.md file and read through it once properly to catch contradictions.

Construction Workflow: From Raw Ideas to CLAUDE.md

Capturing requirements is often the hardest part. A three-step workflow bridges the gap between scattered project knowledge and a well-structured system prompt.

Use a voice-to-text tool (e.g., Wispr Flow) to rapidly dump all requirements. This file is intentionally unstructured.

Ask Claude to restructure the raw ideas into a well-organized markdown document with headings and subheadings. This is NOT the CLAUDE.md yet.

Ask Claude to create the CLAUDE.md file using ideas_v2.md as the source. The CLAUDE.md references ideas_v2.md for full project context.

The unstructured nature of ideas.md would produce a poorly structured CLAUDE.md. The intermediate step ensures proper organization before the system prompt is generated.

Lifespan and Stability

Unlike most project files, CLAUDE.md is not meant to evolve continuously. Its lifespan is weeks to months. It should not change once finalized. The basic ideas and rules it encodes should remain constant throughout the project lifecycle. If something needs to change frequently, it probably belongs in a different file (SKILL.md, feedback files, etc.).

Size Guidelines

CLAUDE.md is not entirely loaded all at once. The most important parts (especially rules) are always loaded. Remaining sections are loaded on demand using keyword matching or RAG.

Split it into persistent memory and dynamic RAG-based retrieval. Keep rules and identity in the always-loaded persistent layer. Move knowledge sections and few-shot examples into retrievable chunks — only what is relevant to the current task gets loaded. This is not a workaround; it is the intended architecture.

Selective Section Retrieval from CLAUDE.md

Not every section of CLAUDE.md is relevant to every API call. The system selectively retrieves sections based on the current task.

Example: A CLAUDE.md with 12 sections totaling ~1,300 tokens. When the task is "write unit test for user service," only 4 relevant sections are retrieved (testing standards, error handling, naming conventions, data validation): consuming ~440 tokens instead of the full 1,300 — a 65% savings.

RAG-Based vs. Keyword-Based Retrieval

In Claude Code, section retrieval is handled automatically by the coding agent. You do not need to implement retrieval yourself. It is mostly done via keyword matching, not full RAG.

Implication for Markdown structure: Headings (#, ##, ###) create natural chunk boundaries. Each section under a heading becomes a retrievable unit. This is why Markdown structure matters: it directly enables selective retrieval.

File Hierarchy: CLAUDE.md, AGENTS.md, and SKILL.md

CLAUDE.md — Four Scoping Levels

CLAUDE.md also supports importing other files using @path/to/import syntax, for example: @README.md, @docs/git-instructions.md, or @~/.claude/my-project-instructions.md.

Strategy: If a preference is universal (web development patterns, coding standards), put it in the global CLAUDE.md. If it is project-specific (writing style for one product), put it in the project CLAUDE.md. Use child-level files for subdirectory-specific rules in monorepos.

AGENTS.md — Sub-Agent Definitions

AGENTS.md is an open standard released by OpenAI in August 2025 and now governed by the Linux Foundation's Agentic AI Foundation (AAIF). It has been adopted by over 60,000 open-source projects and supported by major agent frameworks including Codex, Cursor, Devin, Gemini CLI, GitHub Copilot, Jules, and VS Code.

Key differences from CLAUDE.md:

• AGENTS.md has much more emphasis on specific tools because agents are designed to access tools and execute functions
• AGENTS.md files can be nested in subdirectories to define a hierarchy of agents
• CLAUDE.md defines the overall project; AGENTS.md defines what individual agents within the project do

SKILL.md — Capability Definitions

SKILL.md files define specific skills the model should have for a domain: front-end development preferences, API design patterns, writing style preferences, and more.

Lifespan: Hours to days. SKILL.md files can be heavily modified as more features or specifications are added.

SOUL.md

SOUL.md is a special file (not Claude Code specific) that defines the personality and character of an agent.

The difference between your output when you have a SOUL.md versus when you don't have a SOUL.md is night and day.

— Dr. Sreedath Panat, Course Instructor

Hierarchy Resolution and Override Rules

When multiple AGENTS.md files exist in a nested hierarchy:

1. All AGENTS.md files are loaded and considered
2. If there is a conflict, the deeper (more specific) file's rules override the parent's rules
3. Non-conflicting rules from all levels are merged and used together

Like CSS specificity: more specific selectors override general ones, but non-conflicting styles from all levels apply.

repo-root/
 AGENTS.md <-- Read first (project-wide rules)
 src/
 AGENTS.md <-- Read second (src-specific rules)
 api/
 AGENTS.md <-- Read third (api-specific rules)

Codex assembles: root rules + src rules + api rules
= complete instruction chain for a file in src/api/

Sub-Agents and Context Window Isolation

A critical architectural question: do sub-agents share their parent's context window?

Answer: No. Each sub-agent has its own context window.

Rationale:

• An agent is defined by its ability to make its own API call to the LLM
• Each agent handles its own context window so its actions are not polluted by what is happening globally
• A paper-writing agent has no reason to share context with a coding agent
• Sub-agents make separate API calls and produce outputs based purely on their own context

What IS shared: Certain items like CLAUDE.md are accessible to all agents. Feature specifications may also be shared. But the context window itself is isolated.

Context Lifespan Hierarchy

Different elements of context have vastly different lifespans. The context stack operates like an operating system: Layer 1 is the kernel (always running), Layer 5 is the application (comes and goes).

The more foundational a piece of context is, the longer it should live and the more carefully it should be engineered. CLAUDE.md is the longest-lived and therefore deserves the most deliberate design.

When chatbot performance degrades over a long session, two interventions work: periodically reinject key system prompt rules mid-conversation, or add targeted few-shot examples that directly address the observed degradation.

Few-Shot Examples in System Prompts

Providing examples in the system prompt significantly improves output quality. There are three types of examples based on task complexity.

When a system prompt says "be formal and professional" but the few-shot examples demonstrate a casual, friendly tone, the model follows the examples — not the instruction. Concrete demonstrations of desired behaviour carry higher weight than abstract directives. This means poorly chosen examples can silently override your written rules.

1. Classification Tasks: Input/Output Pairs

The simplest case: a few input/output pairs teach the mapping.

Example 1: [email about meeting invitation] → Primary
Example 2: [email about LinkedIn notification] → Social
Example 3: [email about software update] → Updates

Provide up to 3 diverse examples: one per category if possible.

2. Reasoning Tasks: Chain-of-Thought Examples

When tasks require policy application or multi-step reasoning, the model needs to see how to reason through the problem.

Input: "Customer ordered laptop 45 days ago, wants return"
Reasoning: 45 days > 15-day return window → not eligible
Output: "Return not eligible — outside 15-day return window"

The reasoning step shows the model how to think, not just what to output.

3. Structured Output Tasks: Prefix/Suffix Examples

For tasks requiring specific output formats, provide a complete example of the expected structure:

{
 "order_id": "3301",
 "status": "eligible_for_return",
 "reason": "Within 30-day window",
 "next_steps": ["Initiate return label", "Schedule pickup"]
}

Static vs. Dynamic Example Selection

Dynamic selection outperforms static because only relevant examples are loaded, examples match the current query's domain, and token budget is spent on useful context.

The Diversity Principle

When selecting few-shot examples, the most important criterion is not relevance alone: it is diversity across categories.

• If you have 3 slots: pick 1 return example, 1 shipping example, 1 complaint example — NOT 3 return examples
• Adding more examples beyond 3 has diminishing returns
• Adding non-diverse examples can actually decrease performance because the model overindexes on the repeated category

If three examples are positive, neutral, negative, then adding two more positive examples means you now have three positive, one neutral, and one negative. The diversity suffers and the model becomes biased toward the overrepresented category.

Coding Agents, LLMs, and Platform Configuration

A frequently confused distinction:

CLAUDE.md is not a universal configuration file. It is read exclusively by Claude Code. Other coding assistants do not recognise it. If you switch platforms, you must rewrite your configuration using that platform's equivalent file. AGENTS.md is the closest thing to a universal standard.

Poorly Designed vs. Well-Designed CLAUDE.md

Poorly Designed

• No headings or subheadings (reads like plain text)
• Vague instructions ("design the architecture properly")
• No explicit technology choices
• No project structure
• Could be a .txt file

Well-Designed

• Clear heading hierarchy (#, ##, ###)
• Explicit technology choices
• Project folder structure provided
• Specific rules with concrete guidance
• "Common Mistakes to Avoid" section

The model can learn code patterns and naming conventions by reading source files — but it cannot learn "never use raw SQL here: always use the query builder" or "don't use floating-point for financial calculations." This tribal knowledge (knowledge that exists only in developers' heads and code review comments) is what the anti-pattern section encodes. No amount of code reading will surface it.

Here is an example of a well-designed CLAUDE.md:

# Identity
Customer support agent for ShopCo. Handles return requests,
order status inquiries, and billing disputes via chat.

## Rules
- NEVER approve a return outside the 30-day window
- Always consult the return policy document first
- Damaged items route to escalation queue
- Use snake_case for all variable and function names

## Format
- API responses: JSON with { status, reason, next_steps }
- Build: `npm run build` | Dev: `npm run dev`
- Tests: `npm run test -- --watch`

## Knowledge
- Tech stack: Next.js 14, Tailwind CSS, Supabase, Vercel
- Current phase: Phase 2 (multi-user, launched Jan 2026)
- Return window: 30 calendar days from delivery date

## Tools
- Supabase client for all database operations
- Resend for transactional email
- Sentry for error tracking

## Common Mistakes to Avoid
- Do NOT use raw SQL: always use the Supabase query builder
- Do NOT use floating-point for financial calculations
- Do NOT hardcode the return window (30 days)

Key Takeaways

• System prompt = CLAUDE.md in Claude Code. It is the single most important file you create for any project.
• Five components: Identity, Rules, Format, Knowledge, Tools. Rules are the highest priority and should be the last removed under token constraints.
• Write at the right altitude: not too vague ("be helpful"), not too brittle ("reply: $9.99/month"). Target actionable guidance that remains valid as details change.
• Start minimal, then add: build system prompts iteratively, driven by observed failures, to avoid contradictions.
• Selective retrieval: CLAUDE.md is not fully loaded every call. Markdown headings create natural chunk boundaries for keyword or RAG-based retrieval.
• Few-shot examples significantly improve performance. Three diverse examples covering different categories is the sweet spot.
• Sub-agents have their own context windows — they make separate API calls and their context is not polluted by the parent.
• CLAUDE.md is Claude Code only. Each platform has its own equivalent file. AGENTS.md is the emerging universal standard.
• "Common Mistakes to Avoid" is the highest-value-per-token section because it encodes tribal knowledge the model cannot discover by reading code.
• Context engineering is not physics: it is an evolving art based on thumb rules from community experimentation.

Notes by Isham Rashik · AI Engineer

Based on the Context Engineering course by Dr. Sreedath Panat and Vizuara Technologies.

Version v1.0 · March 29, 2026