From meta-vr
Wires ARKit-blendshape-rigged head/face models to Quest face tracking via Meta Movement SDK (A2E). Use when you have an FBX with 52 ARKit blendshapes and want it driven by facial expressions on Quest Pro / Quest 3 / Quest 3S.
How this skill is triggered — by the user, by Claude, or both
Slash command
/meta-vr:hz-unity-face-trackingThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Quest. Uses the **public** `OVRCustomFace` extension hook — no `OVR_INTERNAL_CODE`, ships to 3P.
End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Quest. Uses the public OVRCustomFace extension hook — no OVR_INTERNAL_CODE, ships to 3P.
The model's blendshape names must follow the ARKit naming convention (camelCase, _L/_R suffixes, e.g. eyeBlink_L, jawOpen, mouthSmile_R). An optional prefix like blendShape2.eyeBlink_L is automatically stripped.
Packages/manifest.json:
com.meta.xr.sdk.core (Meta XR Core — provides OVRFaceExpressions, OVRCustomFace)com.meta.xr.sdk.movement (Meta Movement SDK — A2E + retargeting helpers)Assets/Oculus/OculusProjectConfig.asset (verify via Project Settings → Meta XR):
faceTrackingSupport: 1 (Supported) or 2 (Required)eyeTrackingSupport: 1 if the rig has gazeAssets/Plugins/Android/AndroidManifest.xml) permissions:
<uses-feature android:name="oculus.software.face_tracking" android:required="false" /><uses-permission android:name="com.oculus.permission.FACE_TRACKING" /><uses-permission android:name="android.permission.RECORD_AUDIO" /> (required for A2E)oculus.software.eye_tracking + com.oculus.permission.EYE_TRACKINGOVRManager.FaceTrackingDataSources including Audio (A2E) — if you skip Audio, mouth motion is visual-only.After any change to OculusProjectConfig, call meta_update_android_manifest to regenerate the manifest.
ARKitOVRCustomFace script (in references/ARKitOVRCustomFace.cs) into the project.OVRFaceExpressions component to the OVRCameraRig (or anywhere in the scene).SkinnedMeshRenderer, add ARKitOVRCustomFace. Adding the component triggers Reset(), which auto-populates the blendshape→FaceExpression mapping by scanning the mesh's blendshape names.FaceExpressions field to the OVRFaceExpressions instance.ARKitEyeGazeBlendshapeDriver in (see Eye tracking below).That's the whole flow. Details below.
Don't rely on OVRFaceExpressions.EyesLook* for eye gaze. Those fields are derived from face-camera visuals, not the dedicated eye tracker. On Quest Pro they're often zero or noisy even when face tracking is otherwise working. The right API is OVREyeGaze, which taps the eye tracker directly.
For ARKit rigs with eyeLook* blendshapes (no eye bones), this skill ships ARKitEyeGazeBlendshapeDriver:
OVREyeGaze components (one per eye, TrackingMode = HeadSpace).CenterEyeAnchor).eyeLookUp/Down_{L,R}, yaw → eyeLookIn/Out_{L,R} (with the ARKit "_In = toward nose" convention).LateUpdate, so it overrides whatever ARKitOVRCustomFace wrote in Update.For rigs with eye bones (no eyeLook* blendshapes), skip the blendshape decomposition and just parent OVREyeGaze to each eye bone with ApplyRotation = true — the component will rotate the bone directly.
CenterEyeAnchor (or your head transform): LeftEyeGaze, RightEyeGaze.OVREyeGaze to each. Set Eye = Left / Right, TrackingMode = HeadSpace, ApplyPosition = false, ApplyRotation = true, ConfidenceThreshold = 0.5.ARKitEyeGazeBlendshapeDriver to the head's SkinnedMeshRenderer GameObject (alongside ARKitOVRCustomFace). Wire leftEye, rightEye, and referenceFrame (= CenterEyeAnchor).maxAngleDeg (default 30°) and smoothing (default 0.4) to taste.Copy references/ARKitOVRCustomFace.cs into Assets/Scripts/ARKitOVRCustomFace.cs. It defines the public, 3P-shippable ARKit ↔ OVR FaceExpression table and a MapBlendshapes() method that scans SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i), strips any prefix before the last ., lowercases, and matches against the table. Unmatched mesh blendshapes are set to OVRFaceExpressions.FaceExpression.Max (sentinel — skipped at runtime).
Project settings:
Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported (if needed)
Then regenerate manifest:
meta_update_android_manifestScene Hierarchy
├── OVRCameraRig (from meta_add_camerarig)
│ └── (add) OVRFaceExpressions component
└── YourHeadModel
└── ...SkinnedMeshRenderer GO...
├── SkinnedMeshRenderer (existing)
└── (add) ARKitOVRCustomFace component
└── FaceExpressions = the OVRFaceExpressions ref
└── retargetingType = Custom (set automatically by base when overriding)
└── Mappings[] = auto-filled on Reset()
└── BlendShapeStrengthMultiplier = 100 (default; OVR weights are 0–1, mesh wants 0–100)
If the head has multiple SkinnedMeshRenderers (e.g. separate teeth/tongue meshes), add ARKitOVRCustomFace to each one.
On OVRCameraRig's OVRManager component:
MapBlendshapes() runs automatically when the component is added (Reset()) and from OnValidate() when Mappings is empty. If you need to remap manually (e.g. after re-importing the FBX), use the component's inspector context menu → Map Blendshapes, or from a script:
go.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();
From Unity MCP Unity_RunCommand, the cleanest invocation (the OVR/MSDK types aren't referenced in the MCP dynamic assembly, so use SendMessage to avoid reflection):
GameObject.Find("YourHeadModel")
.GetComponentInChildren<SkinnedMeshRenderer>().gameObject
.SendMessage("MapBlendshapes", SendMessageOptions.RequireReceiver);
ARKitOVRCustomFace component. Mappings.Length should equal SkinnedMeshRenderer.sharedMesh.blendShapeCount. The Console log from MapBlendshapes() reports mapped X/N blendshapes — X should be 50 (or 52 if the FBX has all of them).| Symptom | Cause / fix |
|---|---|
mapped 0/N in Console | Mesh blendshape names don't follow ARKit convention. Verify with mesh.GetBlendShapeName(i). Names must match e.g. eyeBlink_L, jawOpen — case-insensitive, prefix before last . is stripped. |
| Face is frozen | OVRFaceExpressions not assigned, or scene has no OVRCameraRig/OVRManager with face tracking enabled. Check OVRFaceExpressions.FaceTrackingEnabled and ValidExpressions at runtime. |
| Mouth doesn't move when speaking | A2E disabled. Enable Audio under OVRManager → Face Tracking Data Sources, and ensure RECORD_AUDIO permission is granted on device. |
| Eyes don't blink | Mesh's eyelid shapes aren't named eyeBlink_L/_R. Either rename, or add a custom row to ARKitTable. |
| Eyes don't move (look around) | You're relying on OVRFaceExpressions.EyesLook* instead of OVREyeGaze — switch to ARKitEyeGazeBlendshapeDriver. |
| Eye gaze in wrong direction | Note the deliberate ARKit ↔ OVR swap: ARKit's eyeLookIn_L (eye looking nose-ward, i.e. right) maps to OVR EyesLookRightL. The table already does this — don't "fix" it. |
| Weights look half-strength or clamped | BlendShapeStrengthMultiplier defaults to 100 because Unity blendshapes are 0–100 while OVR is 0–1. Don't lower this unless intentional. |
cannot change access modifiers when overriding compile error | Base method is protected internal in another assembly. The override must use protected (not protected internal) — already correct in the supplied script. |
_mappings out of sync with shared mesh assertion at Start | Mesh changed since mapping was generated. Re-run MapBlendshapes() via the component context menu. |
The 52 ARKit shapes don't 1:1 a FACS-based OVR enum. Notable choices baked into ARKitTable:
browInnerUp, cheekPuff, mouthFunnel, mouthPucker, mouthRollLower, mouthRollUpper) pick only the L-side OVR expression. If your model has obviously asymmetric mouth/brow when the user uses these expressions and you want true symmetric drive, subclass and additively sum L+R in a custom MapBlendshapes (use SlothARKitFaceDriver-style per-blendshape sum). For most heads the L-only choice is fine because the face is roughly symmetric and the asymmetry is below visual threshold.eyeLookIn/Out are deliberately swapped per side relative to OVR's left/right semantics (see Troubleshooting).mouthClose → LipsToward (closest FACS analogue).tongueOut → TongueOut (requires HorizonOS ≥ 65 + tongue tracking; otherwise stays at 0).SKILL.md — this filereferences/ARKitOVRCustomFace.cs — drop-in OVRCustomFace subclassreferences/ARKitEyeGazeBlendshapeDriver.cs — OVREyeGaze → eyeLook* blendshape driver (LateUpdate)MonoBehaviour driver that reads OVRFaceExpressions[expr] and writes blendshape weights directly: works, but doesn't integrate with MSDK's correctives, eye constraints, or future retargeting upgrades. Use only if you can't extend OVRCustomFace.OVRCustomFace + RetargetingType.ARKitBlendshapes: clean, but guarded behind #if OVR_INTERNAL_CODE in the public Oculus Integration package — not 3P-shippable.OVRCustomFace + RetargetingType.Custom + GetCustomBlendShapeNameAndExpressionPairs override (this skill): fully public API, shippable, and gets all base-class behavior (data validity gating, weight scaling, mesh assertion).npx claudepluginhub meta-quest/agentic-tools --plugin meta-vrConfigures Meta XR Core SDK for Unity VR/MR projects on Quest headsets. Covers OVRCameraRig, OVRManager, passthrough, hand tracking, spatial anchors, and Android manifest generation.
Adds facial animation and lipsync to characters: phoneme-driven mouth movement and emotional expressions (smile, frown, surprise). Works with Rhubarb Lip Sync, ARKit visemes, and audio from TTS.
Unity 6 XR development guide. Use when building VR, AR, or MR experiences. Covers XR Interaction Toolkit, XR Plug-in Management, OpenXR, hand tracking, controllers, haptics, AR Foundation (plane detection, anchors, image tracking), and platform-specific setup (Meta Quest, Apple Vision Pro). Based on Unity 6.3 LTS documentation.