Tokenspeed

Open source alternative to Fireworks AI, Together AI and NVIDIA NIM

2.7K stars
MIT
Active this week
Last commit 2 days ago

Last verified Oct 20, 2026

Deploy a speed-of-light LLM inference engine built for agentic workloads, with a C++ FSM scheduler, pluggable kernels, and MIT license.

Visit website GitHub2.7K

Read the full breakdownWho it is for, how it works, trade-offs and the FAQ

who it's for

Who Tokenspeed is for#

AI infrastructure teams self-hosting frontier model inference

Teams running inference clusters for models like Kimi K3, DeepSeek V4, or Qwen3.8 on NVIDIA GB200/GB300 or AMD Instinct hardware can deploy TokenSpeed to maximize throughput per GPU. The kernel plugin system allows tuning for specific model architectures and accelerator generations.

Skip if:

Teams without NVIDIA Blackwell or AMD Instinct accelerators, or teams that need a fully managed inference service rather than a self-hosted engine.

Agentic platform developers serving high-concurrency coding agents

Systems generating coding-agent traffic produce contexts over 50K tokens, dozens of conversation turns, and per-user latency requirements around 70-200 TPS. TokenSpeed's scheduler and MLA implementation are designed for this traffic pattern rather than generalized chatbot serving.

Skip if:

Teams building simple chatbots or single-turn applications where standard inference engines already meet latency requirements.

ML researchers and kernel engineers iterating on attention implementations

The pluggable kernel system with its portable public API and centralized registry makes TokenSpeed an environment for writing and benchmarking custom attention kernels. The kernel layer is designed for rapid iteration and includes a documented path for contributing heterogeneous accelerator plugins.

Skip if:

Researchers focused on model training or fine-tuning. TokenSpeed is an inference-only engine with no training path.

Organizations evaluating TensorRT-LLM alternatives for Blackwell deployments

Teams currently using TensorRT-LLM for Blackwell deployments can evaluate TokenSpeed using the published side-by-side benchmarks on Kimi K2.5 SWE-smith traces. The MIT license and non-profit LightSeek Foundation governance differ substantially from NVIDIA's proprietary NIM stack.

Skip if:

Teams with existing TensorRT-LLM deployments meeting performance targets where migration cost outweighs architectural or governance benefits.

the problem

The problem it solves#

Production LLM inference for agentic workloads runs on different constraints than standard chatbot serving. Coding agents generate contexts that routinely exceed 50K tokens and conversations that span dozens of turns, all while users expect fast output, typically 70 tokens per second per user and often 200 or higher. Meeting that latency floor at high concurrency requires careful control over KV cache allocation, batch scheduling, and kernel performance. Most general-purpose inference engines were designed around single-turn or short-context workloads and did not treat these requirements as the primary design target.

A second challenge is hardware heterogeneity. Production inference fleets run on NVIDIA and AMD accelerators, and teams tuning performance for specific frontier models need access to the kernel layer. Closed-source inference services offer no path to kernel-level inspection or modification. Teams serving MoE models like DeepSeek V4 or Kimi K3 on their own infrastructure often find that general-purpose engines leave throughput per GPU unrealized because agentic traffic's KV cache access patterns differ from what those engines optimized for.

how Tokenspeed solves it

How it solves it#

C++ FSM Scheduler with Python Execution Plane

The control plane runs as a C++ finite-state machine, encoding request lifecycle, KV cache ownership, and overlap timing as FSM transitions. Safe KV resource reuse is enforced at compile time through the type system rather than at runtime. The execution plane stays in Python for faster feature iteration and lower cognitive load during research and development.

Pluggable Layered Kernel System

Kernels are separated from the engine core and treated as a first-class modular subsystem with a portable public API, centralized registry, and extensible plugin mechanism. This allows adding or replacing kernels for different accelerators without modifying the engine core, and supports heterogeneous hardware including NVIDIA Blackwell and AMD Instinct.

Optimized MLA for NVIDIA Blackwell

TokenSpeed includes one of the fastest Multi-head Latent Attention implementations for NVIDIA Blackwell for agentic workloads. The decode kernel folds the query-sequence axis into the head axis to improve Tensor Core utilization, and the binary prefill kernel uses a fine-tuned softmax implementation. The TokenSpeed MLA has been adopted upstream by vLLM.

Compiler-Backed Parallelism with Local SPMD

The modeling layer uses a local SPMD design. Developers specify I/O placement annotations at module boundaries, and a lightweight static compiler automatically generates the required collective operations at model construction time, eliminating manual communication logic for tensor parallelism across multiple GPUs.

SMG-Integrated AsyncLLM Entrypoint

Request handling on the CPU side uses an AsyncLLM interface integrated with SMG, providing a low-overhead entrypoint that keeps CPU-side request queueing separate from GPU execution. This is the API surface that external orchestrators, including NVIDIA Dynamo, integrate with.

Day-0 Support for Frontier Models

TokenSpeed has published day-0 launch support for Kimi K3, Kimi K2.5, DeepSeek V4, Qwen3.8, Qwen3.8-Flash-Next, GLM 5.3 Flash, TML Inkling, and Minimax M2.7. It supports both MoE and dense model architectures, NVFP4 quantization, and VLM inference.

strengths · trade-offs

Strengths and trade-offs#

Strengths

Non-Profit Foundation, Vendor-Neutral GovernanceTokenSpeed is developed by the LightSeek Foundation, a non-profit, with hardware and engineering support from NVIDIA, AMD, and model teams. Unlike inference engines built by commercial API providers, there is no incentive to steer users toward a hosted service. The README describes it as the 'Switzerland' of open source inference engines, governed by a non-profit with no commercial hosting service of its own.
Compile-Time Correctness Guarantees in the SchedulerThe C++ FSM control plane enforces safe resource management, including KV cache state, through the type system at compile time. The PyTorch Foundation named this separation of control and execution planes as TokenSpeed's primary architectural differentiator. Correctness is a property of the system structure rather than a convention that runtime checks try to uphold.
Benchmarked Above TensorRT-LLM on Coding-Agent WorkloadsFor Kimi K2.5 on NVIDIA B200, the best configuration (Attention TP4 + MoE TP4) is roughly 9% faster at batch size 1 and roughly 11% higher in throughput at 100 TPS/user for coding-agent workloads above 70 TPS/user. These figures come from LightSeek's published benchmarks using SWE-smith traces, which mirror production coding-agent traffic.
MIT License, No Usage Restrictions for Self-HostersThe MIT license lets teams run, modify, and redistribute TokenSpeed on their own infrastructure without licensing fees, usage metering, or service agreements. There are no commercial restrictions for self-hosted deployments and no dependency on any cloud provider's control plane.

Trade-offs

-Requires High-End NVIDIA or AMD GPU HardwareTokenSpeed targets NVIDIA Blackwell (B200, GB200, GB300) and AMD Instinct (MI355X, MI455X) accelerators. It is not designed for CPU-only inference or commodity cloud GPUs. Teams without access to these accelerator generations cannot run the engine at its intended performance envelope.
-Young Project Still Undergoing Production HardeningDevelopment began in mid-March 2026 and the project was announced in May 2026. As of October 2026, the engine and kernels remain under active development with production hardening ongoing. With 99 open GitHub issues, teams adopting it for production workloads should track upstream changes closely.
-PD Disaggregation Not Yet in the Main DocumentationThe published performance benchmarks focus on single (non-disaggregated) deployment only. Prefill-decode disaggregation support is under development, and the LightSeek team has indicated coverage in a separate follow-up. Teams that need PD disaggregation in production should treat this as a near-term gap.

versus alternatives

Tokenspeed vs alternatives#

TokenSpeed vs Fireworks AI

Both target fast LLM inference, but in different deployment models. Fireworks AI is a managed API with no self-hosting option; TokenSpeed is an open source engine you run on your own GPU hardware.

Feature	TokenSpeed	Fireworks AI
License	MIT	Proprietary
Self-hosting	Yes	No
GPU support	NVIDIA Blackwell, AMD Instinct	Managed fleet
Pricing	Free (your hardware costs)	Per-token API fees
Governance	Non-profit (LightSeek Foundation)	Commercial

TokenSpeed is the better choice when you run your own inference cluster, need kernel-level control for specific models, or serve high-volume agentic workloads where per-token API costs add up. Fireworks AI is the better choice when you need a managed service with SLA guarantees, do not have GPU hardware, or need broad model availability without infrastructure management.

TokenSpeed vs Together AI

Together AI contributed engineering work to TokenSpeed and is listed in its acknowledgments, which makes the two partially complementary. Together AI is a commercial inference API; TokenSpeed is the open source engine co-developed with Together AI's inference team and published for self-hosters.

Feature	TokenSpeed	Together AI
License	MIT	Proprietary
Self-hosting	Yes	No
Control	Full kernel and scheduler access	API surface only
Pricing	Free (your hardware costs)	Per-token API fees

For teams that want the engineering quality behind Together AI's serving stack without the managed API layer, TokenSpeed is the direct path. Together AI remains the better choice for organizations that need managed inference without operational overhead.

TokenSpeed vs NVIDIA NIM

NVIDIA NIM provides containerized LLM inference with NVIDIA's optimization stack. TokenSpeed is an alternative for teams who need MIT-licensed, non-profit-governed inference software on Blackwell hardware rather than NVIDIA's proprietary NIM containers.

Feature	TokenSpeed	NVIDIA NIM
License	MIT	Proprietary NVIDIA
Self-hosting	Yes	Yes (NVIDIA hardware)
Governance	Non-profit (LightSeek)	Commercial (NVIDIA)
Vendor neutrality	NVIDIA + AMD	NVIDIA only

NIM is reasonable when you want NVIDIA's official container support and full optimization stack from the hardware vendor. TokenSpeed is worth evaluating when you need AMD hardware support, MIT license terms, or independent non-profit governance.

tech stack · detected from GitHub

What it's built on#

Languages: C++Python
Frameworks: FastAPI

frequently asked

FAQ#

Is TokenSpeed free to use?

Yes. TokenSpeed is MIT-licensed. You can run it on your own hardware, modify it, and use it commercially without licensing fees or service agreements. The LightSeek Foundation does not offer a managed hosted version; self-hosting on your own GPU hardware is the only deployment path.

How does TokenSpeed compare to vLLM?

TokenSpeed targets higher throughput on agentic workloads specifically, with a C++ FSM scheduler that enforces KV cache resource correctness at compile time rather than runtime. Where vLLM prioritizes broad model compatibility and ease of use, TokenSpeed trades some of that generality for maximum performance on NVIDIA Blackwell and AMD Instinct hardware. The TokenSpeed MLA kernel has been adopted upstream by vLLM, so the two projects share some kernel work.

What hardware does TokenSpeed support?

TokenSpeed has published performance results for NVIDIA Blackwell B200, GB200, and GB300, and AMD Instinct MI355X and MI455X. The kernel plugin system is designed to support heterogeneous accelerators. Older NVIDIA architectures are not listed in the project's performance targets or day-0 model launch records.

What models does TokenSpeed support?

The project has day-0 support records for Kimi K3, Kimi K2.5, DeepSeek V4, Qwen3.8, Qwen3.8-Flash-Next, GLM 5.3 Flash, TML Inkling, and Minimax M2.7. Model recipes covering serving configurations are available in the LightSeek documentation.

Who maintains and funds TokenSpeed?

TokenSpeed is developed under the LightSeek Foundation, a non-profit organization. Hardware and engineering contributions come from NVIDIA DevTech, AMD Triton, Qwen Inference, Together AI, Mooncake, and others listed in the project acknowledgments. The non-profit structure means no commercial hosted inference service is driving the project's direction.

also worth a look

Similar open-source tools#

Vllm

Alternative toTogether AI

High-throughput LLM inference server with OpenAI-compatible API

94,441 GitHub stars
Licence Apache-2.0

LocalAI

Alternative toOpenAI API

Self-hosted AI runtime: any model, any modality, any hardware.

49,798 GitHub stars
Licence MIT

sie

Alternative toBaseten

One self-hosted cluster for all the models your agents need

3,416 GitHub stars
Licence Apache-2.0

Soup

Alternative toTogether AI

Fine-tune any LLM on a 4 GB GPU, one YAML config

9,470 GitHub stars
Licence Apache-2.0

AirLLM

Alternative toHugging Face Inference Endpoints

Run huge LLMs on low-end GPUs with minimal VRAM

39,915 GitHub stars
Licence Apache-2.0

Unsloth

Alternative toTogether AI

Train LLMs locally without code using a browser-based interface

78,315 GitHub stars
Licence Apache-2.0

by category

Open source tools in these categories#

404

4,904 GitHub stars
Licence Apache-2.0
#AI Agents

Omnara

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

2,957 GitHub stars
Licence Apache-2.0
#AI Agents

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

12,712 GitHub stars
Licence AGPL-3.0
#Email

Feature your project

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

TokenSpeed vs Fireworks AI

Both target fast LLM inference, but in different deployment models. Fireworks AI is a managed API with no self-hosting option; TokenSpeed is an open source engine you run on your own GPU hardware.

Feature	TokenSpeed	Fireworks AI
License	MIT	Proprietary
Self-hosting	Yes	No
GPU support	NVIDIA Blackwell, AMD Instinct	Managed fleet
Pricing	Free (your hardware costs)	Per-token API fees
Governance	Non-profit (LightSeek Foundation)	Commercial

TokenSpeed vs Together AI

Feature	TokenSpeed	Together AI
License	MIT	Proprietary
Self-hosting	Yes	No
Control	Full kernel and scheduler access	API surface only
Pricing	Free (your hardware costs)	Per-token API fees

TokenSpeed vs NVIDIA NIM

Feature	TokenSpeed	NVIDIA NIM
License	MIT	Proprietary NVIDIA
Self-hosting	Yes	Yes (NVIDIA hardware)
Governance	Non-profit (LightSeek)	Commercial (NVIDIA)
Vendor neutrality	NVIDIA + AMD	NVIDIA only

Who Tokenspeed is for#

AI infrastructure teams self-hosting frontier model inference

Skip if:

Agentic platform developers serving high-concurrency coding agents

Skip if:

ML researchers and kernel engineers iterating on attention implementations

Skip if:

Organizations evaluating TensorRT-LLM alternatives for Blackwell deployments

Skip if:

The problem it solves#

How it solves it#

C++ FSM Scheduler with Python Execution Plane

Pluggable Layered Kernel System

Optimized MLA for NVIDIA Blackwell

Compiler-Backed Parallelism with Local SPMD

SMG-Integrated AsyncLLM Entrypoint

Day-0 Support for Frontier Models

Strengths and trade-offs#

Strengths

Trade-offs

Tokenspeed vs alternatives#

TokenSpeed vs Fireworks AI

TokenSpeed vs Together AI

TokenSpeed vs NVIDIA NIM

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

LLMOps & AI Tooling

Backend Development

Find open source alternatives to paid software

Who Tokenspeed is for#

AI infrastructure teams self-hosting frontier model inference

Skip if:

Agentic platform developers serving high-concurrency coding agents

Skip if:

ML researchers and kernel engineers iterating on attention implementations

Skip if:

Organizations evaluating TensorRT-LLM alternatives for Blackwell deployments

Skip if:

The problem it solves#

How it solves it#

C++ FSM Scheduler with Python Execution Plane

Pluggable Layered Kernel System

Optimized MLA for NVIDIA Blackwell

Compiler-Backed Parallelism with Local SPMD

SMG-Integrated AsyncLLM Entrypoint

Day-0 Support for Frontier Models

Strengths and trade-offs#

Strengths

Trade-offs

Tokenspeed vs alternatives#

TokenSpeed vs Fireworks AI

TokenSpeed vs Together AI

TokenSpeed vs NVIDIA NIM

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

LLMOps & AI Tooling

Backend Development

Find open source alternatives to paid software