Individual Developer/Researcher
Run powerful DeepSeek V4 Flash or GLM 5.2 models locally on high-end consumer hardware like MacBooks (M-series) or Strix Halo systems, even with SSD streaming for memory-intensive tasks.
Open source alternative to Hugging Face Inference Endpoints, Anyscale Endpoints and Replicate
Last verified
DwarfStar is a small native inference engine optimized for DeepSeek V4 Flash, GLM 5.2, and PRO models on Metal, CUDA, and ROCm hardware.
Run powerful DeepSeek V4 Flash or GLM 5.2 models locally on high-end consumer hardware like MacBooks (M-series) or Strix Halo systems, even with SSD streaming for memory-intensive tasks.
Deploy a multi-user LLM server using existing or older NVIDIA CUDA cards (e.g., L40S) with ds4-server's micro-batching and multi-GPU support, providing cost-effective internal AI capabilities.
Combine the RAM and processing power of multiple systems (e.g., two MacBooks via Thunderbolt) using pipeline or tensor parallelism to run DeepSeek V4 PRO or other large models that would not fit on a single host.
Utilize the native coding agent for low-latency, integrated development and testing of AI agents, leveraging DeepSeek V4 Flash/PRO's tool calling capabilities with persistent sessions.
Running large, capable open-weight language models like DeepSeek V4 Flash/PRO or GLM 5.2 efficiently on consumer-grade hardware or older server GPUs often presents challenges due to memory constraints, performance bottlenecks, and the complexity of setting up specialized inference environments. Users need a highly optimized solution that can leverage available hardware, scale across multiple machines, and provide a seamless experience for local development and deployment.
Specifically tuned for DeepSeek V4 Flash, DeepSeek V4 PRO, and GLM 5.2 GGUF models, ensuring high-quality 2-bit quantizations and efficient execution.
Supports Metal on macOS (M-series Macs), NVIDIA CUDA (including multi-GPU and DGX Spark), and ROCm on AMD Strix Halo systems for broad compatibility.
Enables running models larger than available RAM by streaming routed MoE experts from fast local SSDs, making long contexts practical on machines with less memory.
Offers both pipeline parallelism to sum RAM across multiple systems for larger models and tensor parallelism to split per-layer work between two Macs for reduced latency.
Features an integrated coding agent with low latency, native tool calling, and persistent KV cache sessions, designed vertically for DeepSeek V4 Flash and PRO.
Provides a local HTTP server with OpenAI/Anthropic API compatibility, supporting batched sessions and disk KV caching for multi-user LLM serving.
DwarfStar acknowledges its foundational debt to llama.cpp and GGML, stating that it would not exist without the path opened by the llama.cpp project, including its kernels, quantization formats, GGUF ecosystem, and engineering knowledge. Some source-level pieces are retained or adapted from llama.cpp under the MIT license.
In the context of multi-user LLM serving, DwarfStar highlights its ability to turn servers with older CUDA cards (e.g., Ada Lovelace architecture) into multi-user LLM servers, noting that these cards are "no longer supported for new models by vLLM." This suggests DwarfStar offers an alternative for leveraging hardware that might be phased out by other inference solutions like vLLM for newer models.
make # macOS Metal
make cuda-spark # Linux CUDA, DGX Spark / GB10
make cuda-generic # Linux CUDA, other local CUDA GPUs
make strix-halo # Linux ROCm, AMD Strix Halo
make cpu # CPU-only diagnostics buildDwarfStar is optimized for DeepSeek V4 Flash, DeepSeek V4 PRO, and GLM 5.2 GGUF models. It is not a general GGUF loader and only works with specific, tested GGUF files.
Yes, DwarfStar supports an SSD streaming capacity mode on Metal and for GLM 5.2 on ROCm. This mode keeps non-routed weights resident while streaming routed MoE experts from an in-memory cache loaded from the GGUF file on cache misses.
DwarfStar supports pipeline parallelism to split transformer layers across multiple machines to run larger models, and tensor parallelism to split per-layer work between two GPUs for reduced latency on the same token.
Yes, DSpark speculative decoding is supported for DeepSeek V4 Flash models. It uses an auxiliary draft model to propose future tokens, which are then verified by the main model for faster generation, especially for predictable continuations like code.
The software is currently in beta quality and is undergoing rapid changes. While extensive QA runs are performed before each release, instabilities are definitely possible.
Alternative to
Hugging Face Inference Endpoints
Run huge LLMs on low-end GPUs with minimal VRAM
Alternative to
LM Studio
Run large language models locally on Mac, Linux, or Windows
Alternative to
Grok Bot
Self-hosted AI agent: persistent memory, multi-channel chat, model choice.
Alternative to
Together AI
High-throughput LLM inference server with OpenAI-compatible API
Alternative to
OpenAI API
Self-hosted AI runtime: any model, any modality, any hardware.
Phase-guarded workflow layer for AI coding agents, with eval built in
Run DwarfStar (ds4) on your own server
Self-host DwarfStar (ds4) on a Hostinger VPS.
Deploy on HostingerFeatured
Alternative to
Superhuman
AI email assistant that drafts replies and clears your inbox
Alternative toGrok Bot
AI teammates you own: your keys, your model, your machine.
Alternative toClaude Managed Agents
Open-source agent deployment API. Self-host or use Omnara Cloud.
The projects we add and every new guide. Free, unsubscribe any time.
Claude Code to 
OpenCode
Jira to 
Windshift
Notion to 
AppFlowy
Slack to 
Mattermost
Linear to 
GitLab
Wispr Flow to 
Handy
CapCut to 
Concat
Grok Bot to 
rakazo
Superhuman to 
Inbox Zero
Airtable to 
AITable
Muse to 
OpenClaw
GitHub Copilot to 
Cline
Datadog to 
Netdata
Microsoft Teams to 
Rocket.Chat
Pinecone to 
Qdrant
Zapier to 
n8n
Devin AI to 
OpenHands
Firebase to 
Supabase
1Password to 
KeePassXCConfluence to 
Docmost
Contentful to 
Strapi
Adobe Premiere Pro to 
Shotcut
ChatGPT to 
AnythingLLM
LangSmith to 
Langfuse
