microduck_rl

Open source alternative to NVIDIA Isaac Sim, AWS RoboMaker and Unity Robotics

2.5K stars
Apache-2.0
Active this month
Last commit 10 days ago

Last verified Oct 20, 2026

Build reinforcement learning policies for Microduck's bipedal robot using MuJoCo simulation, then export to ONNX and deploy to physical hardware.

Visit website GitHub2.5K

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who microduck_rl is for#

Robotics researchers studying sim2real transfer

microduck_rl provides a documented, reproducible sim2real pipeline for a real hardware target. The actuator modeling depth (voltage control law, gear backlash, domain randomization) makes it a concrete reference for studying what physical fidelity is necessary and sufficient for successful policy transfer on servo-driven bipeds.

Skip if:

If your research target is a different hardware platform or a larger robot with different actuator characteristics, the environment stack does not generalize without significant rework to the MJCF models and actuator config.

Microduck owners extending the behavior library

The task registry is extensible: each task family is a separate environment config file. Owners who want to teach the robot a new motion can follow the existing task structure, implement reward functions in `mdp.py`, and train with the same pipeline used for the built-in 13 tasks.

Skip if:

If you do not own or plan to own Microduck hardware, there is no general-purpose benefit to this repository over other RL locomotion environments.

ML engineers evaluating RL deployment on embedded hardware

The ONNX export path with a baked normalizer and a shared observation contract demonstrates a concrete approach to deploying learned policies on a resource-constrained robot runtime. Engineers evaluating whether PPO-trained policies can run reliably on embedded hardware will find the export and inference scripts directly applicable.

Skip if:

Skip if your target platform uses a different deployment format (TensorRT, TFLite) or requires a format-agnostic inference pipeline; ONNX is the only supported export path here.

the problem

The problem it solves#

Building bipedal robot locomotion policies requires a simulation environment that faithfully replicates the physics of real servo joints. Off-the-shelf simulators treat actuators as ideal, which means policies that train well in simulation often fail on hardware when they encounter real friction, gear backlash, and voltage sag under load.

For small, low-cost bipedal robots like Microduck, the sim2real gap is especially pronounced. At this scale, tiny servos driving an 800 g frame are the primary source of discrepancy between simulation and physical behavior. Most existing robot RL tooling is built around larger platforms and uses simplified actuator models. Researchers who want sim2real transfer on a platform like Microduck have had to build the actuator modeling and domain randomization stack themselves.

how microduck_rl solves it

How it solves it#

13 locomotion task environments

Covers walking, fall recovery, sit-to-stand, ball kicking, forward rolling (roulade), and roller-skating gaits on flat and rough terrain. Each task has a flat variant and most have a rough-terrain twin; roller tasks add three additional skate-specific environments. The task registry is live and queryable with `uv run list-envs`.

High-fidelity sim2real actuator modeling

Uses the BAM M6 model for the Dynamixel XL330, covering voltage control law, back-EMF, and Coulomb/Stribeck friction. Per-environment domain randomization varies battery voltage, voltage sag under load, command delay, and friction magnitude. A backlash variant adds ±1° of gear play to all 14 servo joints.

ONNX export with baked observation normalizer

The export script bakes the observation normalizer into the ONNX graph, so deployed policies always see correctly scaled inputs without any runtime preprocessing step. The shared 61-dimensional observation contract lets any trained policy hot-swap onto the robot without firmware changes.

PPO training on MuJoCo Warp

Policies train at 50 Hz using PPO through the mjlab framework on MuJoCo Warp. A usable walking gait takes roughly one to two hours at 4,096 parallel environments on a CUDA GPU. Checkpoint resume is one flag away on any train command.

Hugging Face Jobs offload for GPU-free setups

Teams without a local CUDA GPU can route any train command to Hugging Face Jobs by adding the `--hf-jobs` flag. This removes the GPU hardware requirement from the training step and lets the repository run on CPU-only machines for everything except policy training itself.

Hot-swappable runtime policies

Walk, fall-recovery, roulade, and sit-stand policies share a 61-dimensional actor observation layout. Environments that do not use a command slot zero-pad it rather than dropping it, keeping the ONNX export and robot runtime unchanged across the full policy library.

strengths · trade-offs

Strengths and trade-offs#

Strengths

Complete sim2real recipe, not just a task listMost RL environment repos provide task definitions and leave the physics gap to the user. microduck_rl ships the full recipe: BAM actuator modeling, backlash simulation, per-environment domain randomization, and the reward-design lessons from the project documented in AGENTS.md. The gap between training result and hardware deployment is reproducible and explained.
Apache 2.0 license with no usage costApache 2.0 allows commercial use, modification, and redistribution without royalties. Unlike NVIDIA Isaac Sim (proprietary, seat-based licensing) or AWS RoboMaker (pay-per-use cloud compute), the training environment itself carries no licensing fee.
Auditable reward designAGENTS.md documents the reward-design rules distilled from the project, readable by both human engineers and AI coding agents working in the repository. Each task family is a separate environment config file with `ENABLE_*` booleans for domain randomization toggles, so reward functions are visible and modifiable rather than hidden in framework internals.

Trade-offs

-Requires a CUDA GPU for trainingTraining runs through MuJoCo Warp, which requires a CUDA-capable GPU. On ARM-based machines (DGX Spark, Jetson), the first dependency sync pulls around 2 GB of CUDA wheels and can time out; the README documents the `UV_HTTP_TIMEOUT=600` workaround. Hugging Face Jobs offloads this requirement but adds account setup overhead.
-Purpose-built for Microduck hardware onlyThe robot models, actuator configs, and task definitions target the Microduck 14-servo bipedal platform specifically. Adapting these environments to a different robot requires replacing the MJCF models, actuator config, and all joint-layout references throughout the codebase. This is not a general-purpose bipedal locomotion framework.
-3D model files carry a non-commercial licenseThe code is Apache 2.0, but the 3D model files (MJCF exports from Onshape) are Creative Commons BY-SA-NC, which prohibits commercial use. Teams planning to build commercial hardware from the Microduck physical design need to review the CC license separately from the code license.

versus alternatives

microduck_rl vs alternatives#

microduck_rl vs NVIDIA Isaac Sim

NVIDIA Isaac Sim is a full-featured, GPU-accelerated robot simulator built on Omniverse with PhysX physics and broad multi-robot support. microduck_rl is a focused training suite for one 25 cm bipedal platform, built on MuJoCo Warp with an actuator model specific to the Dynamixel XL330 servo.

Feature	microduck_rl	NVIDIA Isaac Sim
License	Apache-2.0	Proprietary (free for research use)
Self-hosting	Yes, requires CUDA GPU	Requires Omniverse/CUDA
Physics engine	MuJoCo Warp	PhysX (Omniverse)
Robot coverage	Microduck only	Multi-robot, extensible
Actuator modeling	Voltage-level BAM model	Simplified joint model

Isaac Sim is the better choice when you need to simulate multiple robot types, integrate with NVIDIA's robotics stack, or use photorealistic rendering for domain randomization. microduck_rl wins when Microduck is your hardware target and you want the exact sim2real recipe, including backlash modeling and BAM actuator physics, at zero licensing cost.

microduck_rl vs AWS RoboMaker

AWS RoboMaker is a managed cloud simulation service built on ROS and Gazebo, with infrastructure provisioning and fleet management. It charges per compute hour and requires an AWS account. microduck_rl is self-hosted, Apache 2.0 licensed, and uses MuJoCo Warp rather than Gazebo.

Feature	microduck_rl	AWS RoboMaker
License	Apache-2.0	Proprietary (pay-per-use)
Infrastructure	Local CUDA GPU or Hugging Face Jobs	AWS cloud (metered)
Physics	MuJoCo Warp	Gazebo
Target platform	Microduck bipedal	Any ROS-compatible robot
Actuator fidelity	High (voltage-level BAM model)	Standard Gazebo plugins

RoboMaker is worth considering when you need fleet simulation at scale with ROS integration and prefer not to manage GPU infrastructure. microduck_rl is the better fit when Microduck is your target, you want zero training cost, and MuJoCo physics fidelity matters more than ROS ecosystem integration.

microduck_rl vs Unity Robotics Hub

Unity Robotics Hub provides tooling for robot simulation in the Unity game engine, used for synthetic data generation and sim2real research with ML-Agents. MuJoCo Warp, which microduck_rl uses, achieves faster physics throughput for RL training at scale than Unity's PhysX-based engine.

Feature	microduck_rl	Unity Robotics Hub
License	Apache-2.0	Proprietary (Unity license)
Physics engine	MuJoCo Warp	PhysX (Unity)
Parallel env support	4,096+ envs on one GPU	Limited by Unity architecture
Actuator modeling	High-fidelity BAM model	Basic joint controllers

Unity Robotics is the better choice when photorealistic rendering, synthetic data generation, or a visual scripting environment matters to your workflow. microduck_rl wins on training throughput, actuator physics fidelity for servo-driven platforms, and licensing cost.

install · quick start

Quick start#

bash

Install via git clone; training requires a CUDA GPU and the uv package manager.
```bash
git clone https://github.com/pollen-robotics/microduck_rl
cd microduck_rl
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096
```

tech stack · detected from GitHub

What it's built on#

Languages: Python

frequently asked

FAQ#

Does microduck_rl require actual Microduck hardware to use?

No. Training and policy visualization run entirely in simulation using MuJoCo. You need a CUDA GPU and the mjlab framework for training, but not the physical robot. Hardware is only required if you want to deploy and test the exported ONNX policy on the real Microduck.

How does microduck_rl compare to NVIDIA Isaac Sim for bipedal RL?

Isaac Sim is a general-purpose robot simulator with broad hardware support, PhysX-based physics, and proprietary licensing. microduck_rl is narrower: it targets Microduck specifically, uses MuJoCo Warp, and is Apache 2.0 licensed at no cost. Isaac Sim is the better choice for teams simulating multiple robot types or integrating with NVIDIA's broader robotics stack. microduck_rl is the better choice when Microduck is your hardware target and you want the exact actuator model and reward design that produced the published sim2real results.

Can I run training without a local CUDA GPU?

Yes. Add --hf-jobs to any train command to submit the run to Hugging Face Jobs instead of running locally. See scripts/hf/README.md for configuration. CPU-only config-invariant and reward-function regression tests are also available via pytest and do not require a GPU.

What is the sim2real gap strategy used in this repository?

microduck_rl addresses the sim2real gap through three mechanisms: BAM actuator modeling (voltage control law, back-EMF, and friction for the Dynamixel XL330), per-environment domain randomization on battery voltage, command delay, and friction magnitude, and explicit backlash simulation adding ±1° of gear play to all 14 servo joints. The observation normalizer is baked into the exported ONNX graph so policies see consistent inputs on real hardware.

Is microduck_rl free for commercial use?

The code and training environments are Apache 2.0 licensed, which allows commercial use, modification, and redistribution without royalties. The 3D model files (MJCF exports) carry a separate Creative Commons BY-SA-NC license, which prohibits commercial use of those assets. If your use case involves the physical robot design rather than just the training code, review both licenses before proceeding.

also worth a look

Similar open-source tools#

REA (Reverse Engineer Anything)

Featured

Alternative toRevEng.AI

AI-powered reverse engineering for binaries, apps, and assemblies.

345,900 GitHub stars
Licence MIT

Omnara

Featured

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

2,957 GitHub stars
Licence Apache-2.0

rakazo

Featured

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

4,904 GitHub stars
Licence Apache-2.0

Agent Reach

Featured

Alternative toFirecrawl

Give any AI agent internet access, free and local

100,165 GitHub stars
Licence MIT

Hermes Agent

Featured

Alternative toGrok Bot

Self-hosted AI agent: persistent memory, multi-channel chat, model choice.

255,514 GitHub stars
Licence MIT

Openfilm

Alternative toRemotion

AI video agent for coding agents; films are local web folders.

1,850 GitHub stars
Licence MIT

by category

Open source tools in these categories#

404

2,957 GitHub stars
Licence Apache-2.0
#AI Agents

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

12,712 GitHub stars
Licence AGPL-3.0
#Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

4,904 GitHub stars
Licence Apache-2.0
#AI Agents

Feature your project

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

microduck_rl vs NVIDIA Isaac Sim

Feature	microduck_rl	NVIDIA Isaac Sim
License	Apache-2.0	Proprietary (free for research use)
Self-hosting	Yes, requires CUDA GPU	Requires Omniverse/CUDA
Physics engine	MuJoCo Warp	PhysX (Omniverse)
Robot coverage	Microduck only	Multi-robot, extensible
Actuator modeling	Voltage-level BAM model	Simplified joint model

microduck_rl vs AWS RoboMaker

Feature	microduck_rl	AWS RoboMaker
License	Apache-2.0	Proprietary (pay-per-use)
Infrastructure	Local CUDA GPU or Hugging Face Jobs	AWS cloud (metered)
Physics	MuJoCo Warp	Gazebo
Target platform	Microduck bipedal	Any ROS-compatible robot
Actuator fidelity	High (voltage-level BAM model)	Standard Gazebo plugins

microduck_rl vs Unity Robotics Hub

Feature	microduck_rl	Unity Robotics Hub
License	Apache-2.0	Proprietary (Unity license)
Physics engine	MuJoCo Warp	PhysX (Unity)
Parallel env support	4,096+ envs on one GPU	Limited by Unity architecture
Actuator modeling	High-fidelity BAM model	Basic joint controllers

Who microduck_rl is for#

Robotics researchers studying sim2real transfer

Skip if:

Microduck owners extending the behavior library

Skip if:

ML engineers evaluating RL deployment on embedded hardware

Skip if:

The problem it solves#

How it solves it#

13 locomotion task environments

High-fidelity sim2real actuator modeling

ONNX export with baked observation normalizer

PPO training on MuJoCo Warp

Hugging Face Jobs offload for GPU-free setups

Hot-swappable runtime policies

Strengths and trade-offs#

Strengths

Trade-offs

microduck_rl vs alternatives#

microduck_rl vs NVIDIA Isaac Sim

microduck_rl vs AWS RoboMaker

microduck_rl vs Unity Robotics Hub

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

Find open source alternatives to paid software

Who microduck_rl is for#

Robotics researchers studying sim2real transfer

Skip if:

Microduck owners extending the behavior library

Skip if:

ML engineers evaluating RL deployment on embedded hardware

Skip if:

The problem it solves#

How it solves it#

13 locomotion task environments

High-fidelity sim2real actuator modeling

ONNX export with baked observation normalizer

PPO training on MuJoCo Warp

Hugging Face Jobs offload for GPU-free setups

Hot-swappable runtime policies

Strengths and trade-offs#

Strengths

Trade-offs

microduck_rl vs alternatives#

microduck_rl vs NVIDIA Isaac Sim

microduck_rl vs AWS RoboMaker

microduck_rl vs Unity Robotics Hub

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

Find open source alternatives to paid software