leanctx

Open source alternative to The Token Company

323 stars
MIT
Active recently
Last commit 60 days ago

Last verified Oct 20, 2026

Optimize LLM token costs with content-aware compression that preserves code verbatim while compressing prose ~40% on-device. MIT-licensed Python SDK.

Visit website GitHub323

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who leanctx is for#

Backend engineers running RAG pipelines with large documents

leanctx routes retrieved passages through LLMLingua-2, trimming each document by roughly 40% before injection into the LLM request. On prose-heavy traffic (retrieved docs, PDFs, meeting transcripts), blended savings reach 36.7%. Code in tool results stays verbatim; only the document text compresses. The N=503 benchmark showed ~$78 saved per 1,000 requests at Sonnet input pricing on a 45.8% prose corpus.

Skip if:

Your retrieved documents are already short or pre-summarized. If average retrieved context is under 500 tokens per request, the compression overhead and the 1.2 GB model download are unlikely to pay off.

LangChain and LangGraph agent developers managing growing tool-call histories

Long-running agents accumulate tool outputs and conversation turns across many requests. leanctx compresses prior conversation history and log dumps while preserving tool_use_id, tool names, and tool inputs byte-for-byte. A 9-message coding agent transcript (~2.1K tokens) showed 35.6% reduction while keeping all tool linkage and code blocks untouched.

Skip if:

Your agent loop is short (under 5 turns) or you already truncate history before each call. For very short conversations, compression overhead does not justify the savings.

LLMOps engineers optimizing production inference costs

leanctx emits OpenTelemetry spans and metrics per call, labeled by provider, method, and status. You can correlate its output with provider-side cache-hit rates in the same dashboard. The SelfLLM cross-provider benchmarks show 41.6-49.1% compression rates using cheap model tiers, at $0.0001-$0.0016 per compression call depending on provider.

Skip if:

Your primary cost driver is output tokens rather than input tokens. leanctx only compresses the request (input) side; it has no effect on generation length or output token counts.

Teams switching away from hosted compression APIs

If your current setup sends prompts to a third-party compression service like The Token Company, leanctx provides an on-premise alternative that runs inside your existing Python deployment with no additional server to operate. Compression latency is 47ms p50 on GPU for the sidecar path, with invariant checks verifying structural integrity before each upstream call.

Skip if:

You need compression for non-Python workloads (Node.js, Go, Java). leanctx is Python-only; the ClawRouter TypeScript sidecar adds server complexity that may outweigh the benefit.

the problem

The problem it solves#

Input token costs become a real budget concern when you are running retrieval-augmented generation with large documents, maintaining long agent conversation histories, or processing multi-step tool call chains that accumulate across turns. A RAG app that injects five 3,000-token documents per query at Sonnet pricing is paying for most of that context on every call, even when the same documents appear across many requests.

The two standard mitigations both have limits. Provider prompt caching covers stable prefixes like system prompts and static document pools, but it does not help when retrieved content is freshly fetched per query. Naive truncation discards context from the middle of documents, which is where many answers live, as the LongBench v2 benchmarks make concrete. Knowing which parts of a live request can survive compression, and which must remain untouched, is harder than shortening text.

how leanctx solves it

How it solves it#

Loss-tolerance routing

Classifies every segment of an outbound LLM request into one of three categories: zero tolerance (code, stack traces, tool_use_id, tool call names and inputs), high tolerance (prose, retrieved passages, conversation history), and conditional (low-confidence prose, oversized context). Each category routes to a different compressor, so structural content is never altered regardless of overall compression settings.

On-device LLMLingua-2 compression

The Lingua route uses LLMLingua-2, an extractive model that runs locally inside your Python process at zero marginal cost per call. On the N=503 LongBench v2 sweep, it compressed eligible prose segments by 40.8% (blended savings of 18.7% on a 45.8% prose corpus). No data leaves your infrastructure during compression.

Drop-in SDK wrapper

Wraps the OpenAI, Anthropic, and Gemini Python clients with the same interface. Switching from `from openai import OpenAI` to `from leanctx import OpenAI` is the only required code change; all existing API call patterns, parameters, and response shapes continue to work unchanged.

Fail-open invariant checks

After compression, leanctx checks message count and ordering, every tool_use_id, tool name and input, and code inside tool results. Any check failure, timeout, or sidecar error sends the original uncompressed request to the provider. A compression outage costs savings, never availability.

OpenTelemetry observability

Emits spans and metrics for every compression call when enabled via `leanctx_config["observability"]["otel"]`. Each root span records provider, method, input and output tokens, cost in USD, and duration. Five labeled metrics (4 counters, 1 histogram) give per-provider and per-method breakdowns without the library owning the OTel SDK.

Reproducible benchmark CLI

Ships seven named scenarios via `leanctx bench run`. Each scenario runs the real LLMLingua-2 model and asserts named invariants, with versioned JSON output. The `agent-structural` scenario checks byte-identical preservation of code blocks, tracebacks, and tool linkage and exits non-zero on any regression, making it suitable as a CI gate.

strengths · trade-offs

Strengths and trade-offs#

Strengths

Prompts never leave your infrastructureUnlike The Token Company and other hosted compression APIs, leanctx runs the compression model locally inside your Python process. No outbound calls go to third-party compression endpoints, and no prompt data is transmitted to external servers. The LLMLingua-2 model loads from a local Hugging Face cache on first use.
Composes with provider cachingleanctx and provider-side prompt caching are complementary. Provider caching covers stable prefixes (system prompts, tool definitions, static document pools) at up to 90% discount on cached reads. leanctx covers the dynamic suffix that changes every call. You mark your stable prefix with cache_control and let leanctx handle the variable portion; both savings stack.
Independently verified benchmarksThe N=503 LongBench v2 sweep was executed and audited by outside contributors, not the maintainer. An earlier draft claiming +7.4% on long context was caught as evaluation noise (McNemar p=0.143), and two methodology fixes followed before results were published. Per-item records are committed to the repo and independently reproducible from a clean checkout.
MIT license, zero marginal cost per Lingua callMIT-licensed with no vendor lock-in. The Lingua route uses the on-device LLMLingua-2 model at zero cost beyond compute; there is no per-call fee and no subscription. The SelfLLM route calls a provider API you configure at your own discretion. No third-party compression billing is introduced.

Trade-offs

-First Lingua run downloads ~1.2 GB of model weightsThe first call through the Lingua route downloads the LLMLingua-2 xlm-roberta-large-meetingbank model weights (~1.2 GB) to the Hugging Face cache. Subsequent calls reuse the cache, but cold-start setup time is meaningful on constrained environments, thin CI machines, or slow network connections. Omitting the `[lingua]` extra leaves leanctx in passthrough mode with no download required.
-Accuracy cost on compressed contentContent routed through LLMLingua-2 in the N=503 LongBench v2 sweep saw a 3.9 percentage point accuracy drop (43.4% to 39.5%). For short/Lingua items specifically (N=68), the drop was 17.6 pp. The overall 1.8 pp hit is bounded by the 54.2% of tokens that route verbatim, but accuracy loss on the compressed half is real and task-dependent. Test on a representative sample of your own workload before enabling in production.
-Python-only SDKleanctx is a Python SDK. Applications written in TypeScript, Go, Java, or other languages have no equivalent in-process client. The ClawRouter integration provides a TypeScript connector via a sidecar architecture, but that adds operational complexity compared to the in-process Python approach.
-Young project with early-stage APIThe repo was created in April 2026, has 319 stars, and 3 forks. It is an early-stage library with a single open issue at the time of writing. The API surface may change across minor releases, and long-term maintenance continuity is not yet established.

versus alternatives

leanctx vs alternatives#

leanctx vs The Token Company

The Token Company is a hosted compression API for LLM prompts. leanctx covers the same use case but takes a different architecture: compression runs inside your Python process rather than on an external server.

Feature	leanctx	The Token Company
License	MIT	Proprietary
Compression location	In-process (local)	Hosted servers
Prompt data leaves your infra	No	Yes
Compression model	LLMLingua-2 (open source)	Closed-source
Self-hosting	Yes (Python package)	No

leanctx is the better choice when your prompts contain sensitive user data that cannot leave your infrastructure, when you need to audit or modify the compression logic, or when you want to avoid per-call billing for a high-volume application. The on-device Lingua route has zero marginal cost per call once the model is cached locally, and the MIT license imposes no resale or usage restrictions on internal team use.

The Token Company is worth considering if you need compression without a Python runtime, want a fully managed service with no model weight download to maintain, or your application is written in a language other than Python. For TypeScript, Go, or Java stacks where a Python sidecar adds operational overhead, a hosted API may be the more pragmatic starting point.

install · quick start

Quick start#

bash

Install leanctx via pip with extras for your LLM provider and the Lingua compression backend.
```bash
pip install 'leanctx[openai,lingua]'
```

tech stack · detected from GitHub

What it's built on#

Languages: JavaScriptPythonTypeScript
Frameworks: FastAPILangChainLangGraph

frequently asked

FAQ#

Does leanctx send my prompts to a third-party compression server?

No. The Lingua route runs LLMLingua-2 locally inside your Python process; the model weights download to your local Hugging Face cache (~1.2 GB on first use). The SelfLLM route calls the LLM provider API you configure, not an intermediate service. No third-party compression endpoint receives your data by default.

How much do token savings vary by workload type?

Savings scale with the prose share of your traffic. Code-heavy agent traffic (25% prose) saves roughly 10.2%; the LongBench v2 benchmark corpus (45.8% prose) saves 18.7%; prose-heavy workloads like document Q&A or transcripts (90% prose) save up to 36.7%. The blended formula documented in the repo is: savings = prose_token_share x 40.8% (95% CI 37.8-43.7%).

What happens if the compression step fails or times out?

leanctx fails open. After compression, it checks message count and ordering, every tool_use_id, tool name and input, and code inside tool results. Any check failure, timeout, or unreachable sidecar sends the original uncompressed request to the provider. A compression outage costs savings, not availability.

Is leanctx compatible with LangChain or LangGraph?

Yes. leanctx wraps the OpenAI, Anthropic, and Gemini Python clients at the SDK level, so any LangChain or LangGraph application using those providers works without additional changes. The project's GitHub topics explicitly list langchain and langgraph as supported integration targets.

How does leanctx compare to provider-side prompt caching?

The two are complementary, not competing. Provider prompt caching (Anthropic cache_control, OpenAI, Gemini) applies up to a 90% discount on stable prefixes like system prompts and static document pools, but it does not help with dynamic per-query content. leanctx compresses that dynamic suffix. Mark your stable prefix with cache_control and let leanctx handle the variable portion; both savings stack independently.

also worth a look

Similar open-source tools#

headroom

Alternative toThe Token Company

Compress LLM context before it reaches the model

76,494 GitHub stars
Licence Apache-2.0

Hermes Agent

Featured

Alternative toGrok Bot

Self-hosted AI agent: persistent memory, multi-channel chat, model choice.

255,514 GitHub stars
Licence MIT

Vllm

Alternative toTogether AI

High-throughput LLM inference server with OpenAI-compatible API

94,441 GitHub stars
Licence Apache-2.0

LocalAI

Alternative toOpenAI API

Self-hosted AI runtime: any model, any modality, any hardware.

49,798 GitHub stars
Licence MIT

Comet

InAI & Machine Learning

Phase-guarded workflow layer for AI coding agents, with eval built in

3,206 GitHub stars
Licence MIT

Orgtree

Alternative toConductor

Desktop canvas for persistent multi-agent AI teams, Windows-only

168 GitHub stars
Licence MIT

by category

Open source tools in these categories#

404

12,712 GitHub stars
Licence AGPL-3.0
#Email

Omnara

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

2,957 GitHub stars
Licence Apache-2.0
#AI Agents

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

4,904 GitHub stars
Licence Apache-2.0
#AI Agents

Feature your project

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

leanctx vs The Token Company

Feature	leanctx	The Token Company
License	MIT	Proprietary
Compression location	In-process (local)	Hosted servers
Prompt data leaves your infra	No	Yes
Compression model	LLMLingua-2 (open source)	Closed-source
Self-hosting	Yes (Python package)	No

Who leanctx is for#

Backend engineers running RAG pipelines with large documents

Skip if:

LangChain and LangGraph agent developers managing growing tool-call histories

Skip if:

LLMOps engineers optimizing production inference costs

Skip if:

Teams switching away from hosted compression APIs

Skip if:

The problem it solves#

How it solves it#

Loss-tolerance routing

On-device LLMLingua-2 compression

Drop-in SDK wrapper

Fail-open invariant checks

OpenTelemetry observability

Reproducible benchmark CLI

Strengths and trade-offs#

Strengths

Trade-offs

leanctx vs alternatives#

leanctx vs The Token Company

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

LLMOps & AI Tooling

Find open source alternatives to paid software

Who leanctx is for#

Backend engineers running RAG pipelines with large documents

Skip if:

LangChain and LangGraph agent developers managing growing tool-call histories

Skip if:

LLMOps engineers optimizing production inference costs

Skip if:

Teams switching away from hosted compression APIs

Skip if:

The problem it solves#

How it solves it#

Loss-tolerance routing

On-device LLMLingua-2 compression

Drop-in SDK wrapper

Fail-open invariant checks

OpenTelemetry observability

Reproducible benchmark CLI

Strengths and trade-offs#

Strengths

Trade-offs

leanctx vs alternatives#

leanctx vs The Token Company

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

LLMOps & AI Tooling

Find open source alternatives to paid software