Things used in this project
Hardware components |
||||
 |
|
× | 1 | |
 |
|
× | 1 | |
|
× | 1 | ||
|
× | 1 | ||
 |
|
× | 1 | |
|
× | 1 | ||
Software apps and online services |
||||
|
||||
|
||||
|
||||
|
||||
 |
|
|||
 |
|
|||
 |
|
|||
Hand tools and fabrication machines |
||||
 |
|
Story
What are you going to buildConcept “Real Hands. Virtual Eyes”
OmniSiege XR is a Mixed-Reality cooperative game that bridges the real and virtual worlds, where two players — one in reality and one in VR — control a single robot together. A VTuber avatar pilot joins from VRChat and steers the vehicle, while a real-world player physically moves a controller arm — and the robot’s arm mirrors those movements exactly. Together, they pick up objects on a game field and deliver them to the goal.
The Arduino UNO Q serves as the brain of the entire system, controlling the vehicle drive system, managing robot arm communication, streaming video to the VR side, and handling game state.
How does your solution work? What are the main features?Attach the Jetson Battery Arm to the robot.
Physical Vehicle:The vehicle uses omni wheels for holonomic movement, allowing it to travel in any direction without rotating. A Jetson handles the drive system on board. A Looking Glass Go lightfield display is mounted on the vehicle to show the VTuber avatar to bystanders in the real world, bridging the virtual and physical presence of the pilot.
Robot Arms (SO-101):Two robot arms are built from the SO-101 kit, which includes all motors and mechanical components. One arm is mounted on the vehicle. The other serves as the real-world controller: the real-world player physically moves it, and the vehicle-side arm mirrors those movements in real time. This allows the real-world player to manipulate objects on the game field through the vehicle.
VR Application:Two implementation paths are supported. The primary target is a Unity standalone VR app built with the XR Interaction Toolkit and Unity WebRTC package, offering low latency. The second option is a custom VRChat world with the robot’s camera feed embedded directly into the scene. In both cases, the VR player’s avatar is a VTuber-style VRM model visible to spectators. Thumbstick input steers the vehicle, and head movement drives the camera pan servo, giving the player a genuine sense of being present on the game field.
Game Flow:The VR pilot steers the vehicle around the field while the real-world player operates the arm to pick up target objects. Together they transport each object to the designated goal position. The team wins by delivering all objects within the time limit.
App Lab Dashboard:A real-time dashboard displays vehicle battery level, arm status, stream latency, and game score, accessible from any browser on the local network.
Will your project focus on sustainable solutions? How?
OmniSiege XR allows players to participate from anywhere in the world over a network, removing the need for travel or a dedicated physical venue. Running game logic and AI entirely on the UNO Q eliminates cloud server dependency and keeps energy consumption low. The vehicle and arm components are designed to be modular, so individual parts can be repaired or replaced independently rather than discarding the whole system. The robot arm outer shells are 3D printed, allowing reprinting of broken parts without replacing the entire assembly.
Briefly explain your user experience.
VR Pilot (VTuber):The pilot puts on a VR headset and joins a VRChat world or Unity app. They immediately see the game field from the vehicle’s camera perspective. Steering is handled by the thumbstick; head movement pans the camera. The experience feels like inhabiting the vehicle from the inside, with the VTuber avatar visible to real-world bystanders through the Looking Glass Go display on the vehicle.
Real-World Arm Operator:The operator holds a physical controller arm — the same SO-101 robot arm — and moves it naturally. The vehicle-mounted arm mirrors every movement in real time. No software knowledge is required; the interaction is as direct as moving your own hand.
The cooperative split — one player navigating from VR, the other manipulating objects through a mirrored arm in reality — is the core UX design. Neither player can complete the game alone, and neither needs to understand the other’s technology to play.
Consider the scalability of your project.
Multiple vehicle-and-arm sets can run in parallel over local WebSocket for event or exhibition play. The VRChat world supports unlimited spectators. The game field layout is fully reconfigurable in software, enabling different difficulty levels, object arrangements, and goal positions without hardware changes. Beyond gaming, the telepresence vehicle and mirrored arm architecture generalizes directly to remote hands-on education, interactive museum installations, and HCI research on cross-reality embodiment and cooperative presence.
How will you use App Lab (Arduino) in your project?
App Lab runs on the UNO Q and connects all parts of the system. The STM32 Arduino sketch handles vehicle motor control via omni wheel drive, reads encoder and sensor feedback, and drives the arm mirror servo commands. Linux Python services host the WebRTC camera stream, the WebSocket game server, and the arm synchronization logic. App Lab ties the sketch and Python side together through the Bridge connection, serves the real-time game dashboard, and provides a configuration panel for setting match duration and goal conditions before each game.
How will you integrate AI in your project?
Two AI modules run on-device on the QRB2210’s Adreno 702 GPU and CPU, with no cloud connection required.
The first is real-time object detection using Edge Impulse. A computer vision model monitors the vehicle’s camera stream and detects target objects on the game field, highlighting them as overlays in the VR pilot’s view. This helps the pilot quickly locate objects and navigate toward them, particularly when the field is cluttered.
The second is adaptive game management. The game server monitors team performance in real time and adjusts object placement difficulty and time limits dynamically, keeping the challenge level appropriate for players of any skill level without manual configuration.
StoryWhat is OmniSiege XR?
OmniSiege XR is a Mixed-Reality cooperative game that bridges the real and virtual worlds, where two players — one in reality and one in VR — control a single robot together. A VTuber avatar pilot joins from VRChat and steers the vehicle, while a real-world player physically moves a controller arm — and the robot’s arm mirrors those movements exactly. Together, they pick up objects on a game field and deliver them to the goal.
The project sits at the intersection of telepresence robotics, VR social platforms, and mixed-reality interaction research. Our goal is to build a system where the boundary between virtual and physical becomes genuinely playable.
Why did we build it?
VTubers and VR social platforms like VRChat have created rich virtual communities — but their connection to the physical world remains mostly visual. We wanted to go further: giving a VR avatar actual physical presence through a robot, and making that presence meaningful through cooperative gameplay. OmniSiege XR is our first step toward that vision, built on an Arduino UNO Q as the core compute platform.
How does it work?The two completed robot arms—one gold and one green.
The system has three main components: two robot arms, a wheeled vehicle, and a VR connection layer.
A 3D print that failed repeatedly
We started by building the robot arms using the SO-101 kit, which includes all the motors and mechanical components needed to assemble two arms from scratch.The outer shells were 3D printed at home in gold ABS and green PLA.After several print iterations and assembly, both arms are now fully operational — capable of gripping and manipulating objects.
One arm is mounted on the vehicle; the other serves as the real-world controller, mirroring its movements to the vehicle-side arm.
The vehicle uses omni wheels for holonomic movement, allowing it to travel in any direction without turning.
A Jetson handles the drive system, and a Looking Glass Go lightfield display is mounted on the vehicle to show the VTuber avatar to bystanders in the real world. The Looking Glass Go is confirmed workingmounting configuration on the vehicle
Running Looking Glass Go on Jetson
We are currently working on integrating VRChat as the VR interface, allowing the pilot to join from any location and steer the vehicle through their avatar. The Arduino UNO Q will serve as the central game controller, managing communication between the vehicle, the robot arms, and the VR side.
——————————————————————————————Update — August 4, 2026:
We completed the design of our bidirectional telepresence architecture.⭐️構成図・アーキテクチャ図（手書きも可）⭐️The system connects two display layers: a Looking Glass Go lightfield display showing the real world to the VR player in stereoscopic 3D, and a point cloud stream rendered inside VRChat showing the real environment to the VR player from the robot’s perspective.
We evaluated four VR platform options — VRChat, Resonite, cluster, and a custom WebXR app — and confirmed VRChat as the onlyder access; cluster was eliminated due to strict external communication limits
We also confirmed that the Looking Glass Go can be driven directly from a PC at 48 viewpoints in a single draw call using WebGL2, with no need for the Looking Glass Bridge software. A survey of prior work confirmed that no existing telepresence system uses lightfield display in this way, making our approach novel.
RealSense Streaming on Jetson Confirmed
We established a stable pipeline for streaming RealSense depth camera data from a Jetson Orin Nano over LAN.
RealSense Video Streaming Test
RealSense Video Streaming Test on Jetson
Due to build issues with pyrealsense2 on ARM64, we switched to a C++ implementation, achieving zero frame drops and 1–2 seconds of end-to-end latency.
Game Rules and Audio Design Finalized
We finalized the core game design: the real-world player operates the robot arm while the VR player steers the vehicle — a cooperative structure that requires both players to succeed.⭐️ゲームフロー図・ルールのメモ書きも可⭐️
The dominant technical challenge is video feedback latency (1–3 seconds), which limits how quickly the VR pilot can react to the results of their commands. We defined three countermeasures: limiting vehicle speed (initial cap: 0.15 m/s), using real-world audio as an instant feedback channel, and keeping vehicle commands discrete and low-speed.
For audio, we decided to mount a smartphone on the robot running VRChat’s mobile app. This requires no additional implementation, eliminates echo at the hardware level, and naturally anchors the voice to the robot’s physical position.
VRChat-to-Robot Drive Bridge Implemented
We implemented vrchat_osc_drive_bridge.py, which reads VRChat avatar OSC output (2-axis puppet), applies dead zone processing and speed scaling, and sends UDP/JSON drive commands at 30 Hz to the vehicle. A watchdog timer ensures the robot stops safely if the connection drops.
Log console of VR control
Camera view from the robot
We also finalized the network layout: robot control runs over a VPN (Tailscale or equivalent) for security, while the camera stream is served publicly. This separates the two channels cleanly by sensitivity.
Update — August 20–23, 2026:
All Three Systems Running Simultaneously (Major Milestone)
For the first time, all three core systems ran stably at the same time: Looking Glass Go display (VR → real world), point cloud stream (real world → VRChat), and VRChat-to-robot drive control.
A robot arm moving in a linked motion
During combined operation, the point cloud feed in VRChat stopped for several seconds every ten to fifteen seconds. We isolated the cause through ablation testing: the Looking Glass rendering was saturating the Jetson GPU (GR3D_FREQ at 98–99%). Reducing the Looking Glass resolution to 240×220 brought GPU usage down to 72–78% and resolved the issue. The key finding is that pixel throughput — not bitrate or network bandwidth — was the bottleneck.
Update — August 23–24, 2026:
UNO Q Integration and Robot Avatar Design
We switched the UNO Q’s object recognition method from HSV color detection to ArUco marker detection. This approach has been verified on hardware and carries the lowest implementation risk within our timeline.
Launch the Arduino app lab, and detect an ARUCO marker with the webcam
We also finalized the VRChat robot avatar architecture. The driver avatar and the robot avatar are separated into two distinct accounts. The robot avatar is controlled by a dedicated bot account drivenchored to the robot avatar’s position using GetPosition() and GetRotation(). This design resolves earlier ambiguity around how to keep the virtual robot model and the physical robot synchronized
Update — August 25–26, 2026:
Drive Control and Avatar Sync Confirmed on Hardware
We confirmed VR controller input driving the physical robot with live camera feed displayed in VR (PASS).
Video display was switched from a custom OpenVR overlay to SteamVR’s window pinning feature, after the overlay API returned errors on our hardware.
We also resolved a VRChat SDK issue: the TMP InputField “On End Edit” event does not fire in the actual VRChat client (a known bug). We replaced it with a Button OnClick() handler as a workaround. The point cloud follow UI is now confirmed working.
Current State — August 28, 2026Current overall picture
- Robot drive avatar: VR controller input → physical robot control → VRChat avatar sync confirmed. Minor discrepancy between physical and avatar movement under investigation.
- UNO Q WS2812B LED control: confirmed on hardware.
- UNO Q ↔ VRChat game state sync: not yet started.
- VRChat world game logic (prompt display, timer, scoring): not yet started.
- Full run-through rehearsal: not yet conducted.
We are documenting the build process as we go and will continue updating this page with new progress, videos, and technical details.
Update — Late August to Early September 2026: Drive Bridge Scripts Implemented
We implemented a two-client bridge architecture separating the VR-side PC (vehicle control) from the real-world PC (arm control).bridge_teleop.pyruns on the VR-side PC and merges vehicle drive commands from VRChat OSC with arm data from the real-world PC via UDP, sending the combined output to LeKiWi.arm_sender.pyruns on the real-world PC and streams SO-101 leader arm joint angles in real time. Both scripts include watchdog timers that bring the vehicle to a safe stop within approximately 0.5 seconds if the connection drops, confirmed on hardware.
Update — September 6–8, 2026: UNO Q ↔ VRChat Communication Resolved
We initially attempted UDP-based communication between the UNO Q and the VRChat game state relay, but discovered that the App Lab Docker container environment does not expose arbitrary UDP ports. We switched to a poll-based approach in which the UNO Q periodically sends HTTP POST requests to the Watcher PC’s/syncendpoint at approximately 300 ms intervals. End-to-end operation was confirmed, removing the last communication blocker between the UNO Q and VRChat.
Update — September 8–9, 2026: UNO Q MCU Hardware Failure
During testing of the WS2812B LED ring, repeatedly connecting and disconnecting the external power supply while the data line remained attached to the UNO Q caused physical damage to the STM32U585 MCU’s peripheral I/O circuits (UART/I2C). We have submitted an RMA inquiry to Arduino’s official support and are awaiting a response.
The impact on the game is limited. Arm mirroring, ArUco recognition, the HTTP API, and Gist sync all run through separate paths that do not depend on the MCU. We confirmed in code that MCU communication failures are caught and logged without halting game execution, meaning the game can proceed without LED effects.
Update — September 10, 2026: Four VRChat World Features Completed
We completed four previously outstanding VRChat world implementations: player seat locking (VRC Station + Immobilize to prevent the pilot’s avatar from drifting during thumbstick input), match result display UI, Spout2 capture camera position alignment (using in-world marker display rather than manual numeric entry), and a simplified game field stage model.
Update — September 13, 2026: Direction A Video Pipeline Complete and Full Run-Through Rehearsal
We completed the Direction A video pipeline, which streams the VRChat virtual camera feed to the Looking Glass Go for stereoscopic display on the robot. The pipeline runs VRChat (CaptureCamera with RenderTexture + RGBD encoding shader + Streaming Camera) → OBS Spout2 capture → NVENC (B-frames set to 0 for WebRTC compatibility) → mediamtx → WHEP → Looking Glass Go, and was confirmed working end-to-end on hardware.
VRChat XR game world
Score Board on the VRC world
On the same day we ran the first full system rehearsal and confirmed that the overall game flow operates as intended.
Current Status — September 13, 2026
- Drive bridge scripts (bridge_teleop.py / arm_sender.py): confirmed on hardware
- UNO Q ↔ VRChat communication (poll method): end-to-end confirmed
- VRChat world features (seat lock, result UI, stage, camera): complete
- Direction A video pipeline (VRChat → Looking Glass Go stereoscopic): confirmed on hardware
- Full run-through rehearsal: completed — overall operation confirmed
- UNO Q MCU: hardware failure; LED effects unavailable (no impact on game flow)
Known Issues — as of September 13, 2026
- Video stream degradation over time (frame drops and latency increase with duration). Mitigation plan: enforce LAN-only connection for the event and reduce resolution and bitrate by one step.
- Game start event not propagated as a Networked Event to the robot avatar PC, causing the UNO Q to miss the start signal. Under active investigation as the highest priority fix.
- Rotation calibration incomplete — the multiplier for compensating angular speed still needs to be tuned against measured robot rotation.
- Game task count reduced from 3 to 1 delivery objective to improve playability within the time limit.
Schematics
File missing, please reupload.
Credits
Revetronique
1 project • 0 followers
We’re an independent researcher & media artist duo. We explore the theme “embodying philosophy and pursuing truth through Art & Technology.”
ContactContact
1 Comment
Pingback: 15 TV Shows That Predicted The Future - Comic Vibe