- RAYFRONTS // IROS 2025
- OPEN-SET LABELS
- FRONTIER EXPLORATION
NEXUS
THE CONNECTING POINT
Robots operating in unstructured environments need to understand what they see. NEXUS solves open-set semantic mapping: label arbitrary objects on-the-fly, build consistent 3D representations, and explore efficiently to maximize information gain. Built on RayFronts (IROS 2025).
MODULE STATUS: ACTIVESemantic Encoding
OPEN
- DIVISION
- ANIMA
- WAVE
- W3
- DOMAIN
- UNDERSTANDING
- WAVE 3 // ANIMA SUITE
- OPEN-SET SEMANTIC MAPPING
ROBOTS CAN'T NAME WHAT THEY SEE
Robots operating in unstructured environments need to understand what they see. Off-the-shelf object detectors only recognize pre-trained classes — they fail on novel objects, rare materials, and task-specific entities. This forces expensive manual annotation.
Building persistent, searchable 3D maps requires both geometry (where is everything?) and semantics (what is everything?). Current systems treat these as separate problems, leading to fragmented representations. NEXUS unifies them.
WHAT NEXUS DELIVERS
NEXUS is a containerized semantic mapping and exploration service built on RayFronts (IROS 2025). Maps semantically, explores intelligently, processes online.
PIPELINE
- 01Maps semantically — assigns open-set labels (any category) to 3D surfaces in real-time
- 02Explores intelligently — frontier-based, entropy-driven, and information gain strategies
- 03Processes online — updates maps incrementally as new sensor data arrives
- 04REST and gRPC APIs — designed for the ANIMA manipulation pipeline
CAPABILITIES
- MULTI-RUNTIMECPU, GPU, and Apple Silicon (MLX M1–M5) runtimes→ ~60ms on M5, 4× M1 throughput
- FLEXIBLE MODELSDeterministic baseline encoder + optional CLIP for custom vocabularies→ No retraining required for new object categories
- ADMISSION CONTROLBounded admission control for concurrent inference requests→ Production-grade reliability under load
WHY THIS IS HARD
Open-set recognition without task-specific training requires grounding visual embeddings to arbitrary language:
- 01CLIP integration for custom vocabularies requires careful offline caching and fallback
- 02Frontier discovery and entropy-driven viewpoint selection when the semantic space is unbounded
- 03gRPC service contracts, protobuf schema versioning, health checks, and bounded admission control
- 04Real-time latency constraints for robotic systems — no batch processing allowed
- 05Cross-team integration across REST and gRPC with consistent protobuf schemas
NEXUS wraps all this behind clean async APIs with deterministic baselines and optional CLIP enhancement.
REAL HARDWARE PERFORMANCE
Measured with RayFronts encoder + semantic pipeline:
| METRIC | VALUE |
|---|---|
| Semantic Encoding | ~60ms (M5 MLX), ~25ms (RTX 4090) |
| Apple M5 (MLX) | ~60ms per frame (4× M1) |
| Map Update Rate | Real-time incremental (no batch) |
| Exploration Strategies | Frontier, entropy, information gain |
| API Protocols | REST + gRPC |
| Admission Control | Bounded queue per deployment |
| Model Cache | Offline-safe with mounted volumes |
WHAT'S BUILT TODAY
| COMPONENT | STATUS | NOTES |
|---|---|---|
| Core Models | COMPLETE | RayFronts encoder + semantic pipeline, production-ready |
| API Layer | IN PROGRESS | Public API, pending dataset infrastructure |
| REST API | COMPLETE | FastAPI with /docs auto-gen |
| gRPC Service | COMPLETE | Protobuf contract + reflection |
| Semantic Baseline (MLX) | COMPLETE | Deterministic, no network calls |
| CLIP Integration | COMPLETE | Local offline cache, offline-safe |
| Exploration Strategies | COMPLETE | Frontier, entropy, information gain |
| Admission Control | COMPLETE | Bounded queue per deployment |
| Quality Gates | COMPLETE | ruff, pytest, mypy passing |
WHERE NEXUS DEPLOYS
- APP_01
ROBOTIC INTEGRATORS
Building SLAM + semantic understanding stacks. Plug NEXUS in for open-set object labeling on top of geometric maps.
- APP_02
MANIPULATION TEAMS
Scene graphs for grasping and object search. Know what's on the table before reaching for it.
- APP_03
AUTONOMOUS EXPLORATION
Search and rescue, inspection drones. Explore unknown environments and build semantic maps in real-time.
- APP_04
COLLABORATIVE ROBOTS
Sharing maps across heterogeneous agents. One robot maps, all robots understand.
- APP_05
WAREHOUSE AUTOMATION
Semantic inventory mapping and search. Find any object by name, not by coordinates.
- APP_06
SMART BUILDINGS
Object-level understanding for facility management. Track assets, monitor spaces, automate responses.
UNDER THE HOOD
FOUNDATION: RAYFRONTS
- Open-set semantic mapping from egocentric streams (IROS 2025)
- Frontier-based exploration with global loop closure
- MIT-licensed reference implementation
- Deterministic baseline encoder with optional CLIP
NEXUS IMPLEMENTATION
- FastAPI REST Server
- Pydantic schemas with /docs auto-generation
- gRPC Service
- Protobuf 3 contract with reflection
- Device Abstraction
- MLX (Apple Silicon) → CUDA → CPU fallback chain
- Model Caching
- Offline-mode fallbacks with mounted volumes
- Admission Control
- Bounded queues for concurrent request management
INTEGRATION POINTS
- Feeds 3D semantic maps to SYNTHESIS (collaborative SLAM)
- Provides object labels to TACTIS (vision-touch fusion)
- Supplies scene context to downstream manipulation modules
RESEARCH BASIS
- [01]RayFronts — open-set semantic mapping for production robotics (IROS 2025)