Headroom: Context Compression Before the LLM

You build an agent.

It reads tickets, pulls logs, calls tools, collects JSON, and adds a few trace fields. All correct. All somehow relevant.

Then the next LLM call goes out.

20,500 tokens.

And the absurd part is: the model may only need 6,700 of them. The rest is repetition, debug noise, and structured bulk that feels expensive but does not make the system smarter.

That is why Headroom is interesting.

Not as a new agent framework. Not as a prompt trick. As a local context compression layer directly before the LLM.

In the OpenAI test from the video, the context drops from about 20,500 tokens to around 6,700 tokens. Almost 67 percent less input. Same answer quality. And, depending on the integration mode, without rebuilding your app.

AI agents rarely fail because the model gets too little text.

They fail because the context is full of things that are technically correct, but barely useful for the next decision: complete tool outputs, repeated log lines, huge JSON blocks, almost identical error messages, and metadata that changes slightly on every call.

That costs you three times:

• Money, because input tokens are billed linearly.
• Latency, because larger prompts take longer to process.
• Quality, because relevant information gets buried in noise.

In the video#

The video is the short, practical version:

• 00:00 — The problem: why your agent eats too many tokens
• 00:44 — What is Headroom? Intro and overview
• 01:45 — Architecture: how Headroom is built
• 04:20 — Python notebook: live demo with before/after comparison

Links:

• Headroom: github.com/headroomlabs-ai/headroom
• Notebook demo: github.com/XamHans/headroom-demo

The problem: Context is not storage#

When you build an agent, it is tempting to put everything into the next LLM call.

The agent read a ticket? Put it into context.

It fetched logs? Put them in.

It has tool responses, trace data, JSON, status, and metadata? Put those in too.

At first, this works. Then the pipeline grows. A few more tools. A few longer outputs. A few debug fields.

Suddenly you are sending 20,000 tokens even though the model may only need 5,000.

The problem is not just the context limit. The problem is attention.

Every token competes with every other token.

The idea behind Headroom#

Headroom is a local context compression layer for AI agents.

It sits between your application and the LLM:

Inputs
  -> Headroom compress
  -> LLM
  -> Metrics

Headroom does not try to creatively summarize the content.

That would be dangerous. Once a compression layer starts interpreting, you have created a new failure mode.

Instead, Headroom does something more boring and more useful:

• Detect repetitions.
• Compress structured data.
• Remove low-value redundancy.
• Preserve unique information.
• Manage the context window deliberately.

The goal is simple: fewer tokens in, same signal out.

Headroom runs locally before the LLM and compresses context before it gets expensive.

The pipeline#

Headroom consists of three components.

1. Cache Aligner#

Caching works best when the stable part of the prompt stays stable.

Many agent prompts have dynamic information in the middle of the prompt: timestamps, IDs, status values, small tool results. That means the prompt changes early, and the cache hit rate gets worse.

The Cache Aligner moves dynamic parts closer to the end and keeps the static prefix more stable.

That sounds small, but it is an important infrastructure decision. Prompt caching is not just a provider feature. It is something you have to actively design your context for.

2. Smart Crusher#

The Smart Crusher is the actual compression logic.

It looks for patterns that are obviously redundant for both humans and models:

• identical log lines
• repeated stack trace fragments
• JSON arrays with the same structure
• repeated metadata
• long values that add no extra information for the task

Important: the goal is not to make everything as short as possible. The goal is to improve the signal-to-noise ratio.

If a piece of information is unique, it stays. If 200 log lines show the same pattern, the model does not need to see all 200 of them.

3. Context Manager#

The Context Manager is the safety line.

It watches how full the context window gets and decides what to prioritize when things get tight. Compression alone is not enough. You need a policy:

• What is task-critical?
• What is only debug noise?
• What can be shortened?
• What must stay verbatim?

Without this layer, context compression quickly turns into blind trimming. And blind trimming is just another kind of bug.

Three ways to use Headroom#

The important bit: Headroom is not trying to be a framework that takes over your entire agent architecture.

That matters. Good infrastructure layers solve one narrow problem and leave your pipeline alone.

The notebook shows the three modes clearly:

1. compress() directly in code#

This is the cleanest path when you control the agent pipeline yourself. You build your messages, call compress(), and pass result.messages to the LLM client.

from headroom import compress
 
result = compress(
    messages,
    model="gpt-4o",
    compress_user_messages=True,
    protect_recent=0,
    target_ratio=0.3,
    min_tokens_to_compress=0,
)
 
compressed_messages = result.messages
print(result.transforms_applied)

This is the mode from the notebook demo because it makes the before/after comparison the most visible.

2. Proxy#

The proxy is interesting for existing systems: Headroom runs as a local intermediate server in front of your provider.

headroom proxy --port 8787

Your app then talks to the local proxy instead of directly talking to the LLM endpoint. This is the option when you want to touch as little code as possible.

3. SDK or wrapper#

The third path is a wrapper around existing clients or agent tools. The Headroom repo describes this mode as wrap for coding agents and as an SDK for custom integrations.

headroom wrap claude
headroom wrap codex

Or inline in TypeScript/Python code if you want to control compression explicitly at the call site.

The pattern is always the same: Headroom runs locally, compresses the context, measures the result, and passes the compressed input forward.

What happens in the notebook#

The demo notebook builds an artificial but realistic incident context: payment logs, support tickets, and root-cause notes.

The raw context intentionally looks like what agents constantly see in production: lots of repetitive logs, large JSON arrays, and a few facts that actually matter.

def build_incident_context():
    incident_logs = "\n".join([
        f"2026-06-23 10:14:{i:02d} payment-service ERROR "
        f"checkout_id=chk_{1000+i} provider=stripe status=502 "
        f"retry={i%3} message=upstream timeout from payments-api"
        for i in range(120)
    ])
 
    support_tickets = json.dumps([
        {
            "ticket_id": f"CS-{2000 + i}",
            "customer_tier": "enterprise" if i % 4 == 0 else "pro",
            "channel": "chat" if i % 2 == 0 else "email",
            "issue_type": "payment_failed",
            "status": "waiting_for_retry",
            "country": ["DE", "AT", "CH", "NL"][i % 4],
            "message": "I tried to pay and got an error. Please check my invoice.",
        }
        for i in range(180)
    ], indent=2)
 
    return [
        {"role": "system", "content": "You are helping investigate a payment incident. Keep only high-signal facts."},
        {"role": "user", "content": "Please analyze the incident material and summarize what matters for the on-call response."},
        {"role": "tool", "tool_call_id": "logs", "content": incident_logs},
        {"role": "tool", "tool_call_id": "tickets", "content": support_tickets},
    ]

This is a good test bed because it simulates exactly the kind of context that gets expensive in agents. Logs and tickets matter, but most of the bulk is repetition.

Then the notebook measures the raw context, compresses it, and compares the token counts:

messages = build_incident_context()
raw_tokens = token_count(messages)
 
result = compress(
    messages,
    model=MODEL,
    compress_user_messages=True,
    protect_recent=0,
    target_ratio=0.3,
    min_tokens_to_compress=0,
)
 
compressed_tokens = token_count(result.messages)
tokens_saved = raw_tokens - compressed_tokens
saved_pct = (tokens_saved / raw_tokens * 100) if raw_tokens else 0
 
print(f"SAVED TOKENS: {tokens_saved}")
print(f"SAVED PCT: {saved_pct:.1f}%")
print(", ".join(result.transforms_applied))

The important part: Headroom does not compress blindly. It returns which transformations were applied. That matters for debugging. An optimization you cannot observe will eventually become dangerous in production.

Live comparison against OpenAI#

The notebook sends the same task to OpenAI twice:

1. once with the raw context
2. once with the Headroom-compressed context

The comparison uses the real prompt_tokens from the API response, not just a local estimate.

def call_openai(label, context_text):
    msgs = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"{context_text}\n\n{QUESTION}"},
    ]
    t0 = time.perf_counter()
    response = client.chat.completions.create(
        model=model,
        messages=msgs,
        max_tokens=150,
    )
    latency_ms = (time.perf_counter() - t0) * 1000
    u = response.usage
    return {
        "label": label,
        "latency_ms": round(latency_ms),
        "prompt_tokens": u.prompt_tokens,
        "completion_tokens": u.completion_tokens,
        "total_tokens": u.total_tokens,
        "reply": response.choices[0].message.content,
    }

This is the part that matters most to me. Local token estimates are useful, but in the end, what matters is what the provider bills and whether the answer still works.

In the OpenAI test from the video, the prompt goes from about 20,500 tokens down to around 6,700 tokens. That is almost 67 percent fewer tokens. The measured latency in the same test drops from 6 ms to 1 ms.

The Headroom repo also shows workloads with 92 percent savings, for example Code Search and SRE Incident Debugging. I would intentionally read those numbers separately: 67 percent is the OpenAI practice test shown in the video, 92 percent is a more compressible benchmark from the repo.

That is not a promise that every pipeline saves 92 percent. It is a signal: in real agent pipelines, there is often a lot of headroom before you even switch models.

Output tokens: The other half of the bill#

Headroom does not only reduce input. The notebook also includes a small demo for output-token reduction.

The point: with many models, output tokens are much more expensive than input tokens. And models like to write things back that you do not need:

• preambles like "Sure, here is..."
• repeating the question
• printing the context again
• deep explanations for routine steps

The notebook simulates simple verbosity steering:

VERBOSITY_NOTE = (
    "\n\nBe direct and concise. Do not restate the question, repeat context already "
    "provided, add preambles like 'Great, let me...' or 'Certainly!', or include "
    "closing summaries unless explicitly asked."
)
 
normal = call("NORMAL", BASE_SYSTEM)
shaped = call("VERBOSITY STEERING", BASE_SYSTEM + VERBOSITY_NOTE)
 
saved = normal["completion_tokens"] - shaped["completion_tokens"]
saved_pct = saved / normal["completion_tokens"] * 100

That is not the core of Headroom, but it shows the larger mental model: token costs happen on both sides of the call. Good AI infrastructure optimizes input and output.

Why this gets more important#

Many teams optimize AI systems in this order:

1. bigger model
2. better prompt
3. more context
4. even more context

That is understandable, but expensive.

A better reflex is:

1. What does the model actually need?
2. What is redundant?
3. What can be compressed structurally?
4. What must remain unchanged for quality reasons?
5. How do we measure tokens, ratio, and latency per call?

Headroom is an answer to exactly these questions.

Not as a magic optimization. As an engineering layer.

The actual mental model#

Context compression is not a prompt trick.

It is infrastructure.

When you build agents, context is not just text. It is your runtime state.

And runtime state has to be managed: sorted, prioritized, measured, compressed, and protected.

Headroom makes that layer visible.

Not every token is equally valuable. Good AI infrastructure does not treat it that way either.