hz-unity-passthrough-camera-access

Meta Quest Passthrough Camera Access (PCA)

Building on top of the Android Camera2 API, Passthrough Camera Access provides access to the forward-facing RGB cameras on Quest 3 and Quest 3S for the purpose of supporting Computer Vision and Machine Learning. This API is distinct from the Media Projection API, which supports casting from the user's POV (including the UI) and should be used if your purpose is to represent what the user is seeing. Use the Passthrough Camera API to add application-specific computer-vision capabilities that extend the understanding of the user's environment and actions beyond what is provided by the Quest Scene API.

Use Cases

This API provides an unobstructed view using the forward-facing RGB cameras and can be integrated with ML/CV pipelines. Common use cases:

1. Specially-trained ML/CV models that identify specific objects the application can interact with (e.g. fitness equipment like dumbbells, industrial equipment like audio mixers).
2. ML/CV assistants or guides for experiences. For instance, a museum tour where looking at a painting surfaces information about its history, artist, and style using off-device LLMs.
3. Feedback for training or special-interest applications — after asking the user to perform a task, the app can detect whether it was done correctly by interpreting changes to the environment (e.g. writing on a whiteboard).
4. Design improvements. By interpreting the lighting or using the camera image to modify a texture, much more realistic design effects can be achieved.

General Prerequisites

1. Horizon OS v74 or later.
2. Quest 3 or Quest 3S (Quest Pro and earlier are not supported).
3. Either permission android.permission.CAMERA or horizonos.permission.HEADSET_CAMERA. CAMERA grants access to both the passthrough and avatar cameras; HEADSET_CAMERA grants access only to the passthrough camera.
4. The Passthrough feature must be enabled to access the Passthrough Camera API.

Passthrough Camera Using Android Camera2

Camera Access on Quest is implemented on top of Android's Camera2 API within Horizon OS. Starting from Horizon OS v74, Camera2 and its Unity extension are available on Quest headsets (Horizon OS v83 added an Unreal extension). On Quest 3 and Quest 3S developers have access to the left and right cameras on the face of the HMD.

The Android Camera2 API provides:

• Image capture: capture camera data for advanced processing.
• Camera metadata: query the API for hardware capabilities and configuration information.
• Multi-camera support: access and control multiple cameras.

Known Issues

1. After the first install of an app using the Camera API in v74 after flashing a new OS, the RGB channels might flip causing color distortion (e.g. flesh tones appear bluish). Reboot the device to fix.
2. Passthrough Camera API is not supported in XR Simulator.
3. The passthrough camera texture captures a rectangular area (1280×960) smaller than what the user sees. The 1280×1280 resolution added in v83 expands the vertical field of view, but still does not cover the entire passthrough view.
4. Restrictions applied by a parent to restrict Teen and Youth accounts (e.g. accessing the Passthrough Camera API) are not applied if the application is installed through the Meta Quest Developer Hub.

Best Practices

1. Ensure user privacy. Camera image data is considered Device User Data and is covered by the Developer Data Use Policy. Adhere to this policy fully.
2. Avoid costly processing on device. On-device processing of camera images can enable compelling use cases, but keep experiences comfortable by maintaining a high framerate.

References

Detailed, task-specific guidance for projecting and placing PCA imagery in world space lives under references/:

• references/principal-point-offset.md — Correctly project the camera image onto a flat world-space surface (frustum-slice quad, FOV visualizer, image-plane overlay) using the optical axis as the plane normal. Read this when corner rays from ViewportPointToRay produce a non-rectangular quad, a camera-aligned quad drifts off-center, or 2 * d * tan(fov / 2) gives the wrong size on Phoenix/Stanley but worked on Quest 3.
• references/detection-bounding-boxes.md — Draw precise 3D bounding boxes around real-world objects detected by an ML model. Read this when placing 2D ML detections (YOLO, SSD, DETR, custom CNN) into world space, building a camera-facing billboard quad from a 2D detection, or wiring PassthroughCameraAccess + EnvironmentRaycastManager for object localization.