Computer Vision for Meta Quest 3

A mixed-reality computer-vision system that connects a Meta Quest 3 application to a Python processing pipeline for marker detection, camera calibration, pose estimation, and distance measurement in a biomedical research context.

Project details

• Built the headset side as a Unity mixed-reality application using Meta's passthrough camera API, Oculus configuration, Android permissions, and XR tooling.
• Created a custom stream by compositing real-world passthrough pixels with Unity-rendered virtual objects through a separate off-screen camera.
• JPEG-encoded the composite frames and sent them from the Quest to a PC over TCP, with processed results returned to the headset for display.
• Implemented a Python workflow for live stream preview, calibration-image capture, camera calibration, and real-time 6x6 ArUco marker detection and pose estimation.
• Computed marker distance and forward depth from the estimated pose and fed compact measurement results back into the headset UI.
• Dockerized the PC-side workflows while keeping the Unity/Quest application in the native Unity and Android toolchain.
• Developed the project as part of a wider biomedical research project at the SYstems and Materials for MEchatronics (SYMME) laboratory.

Technologies