Head Tracking Scenes

Occluders

Occluders are visual elements that represent real objects in a scene and are used to hide parts of AR objects behind them. They are not rendered themselves but participate in depth test, in other words they only signal not to draw parts of meshes behind them being completely transparent at the same time. For example in face tracking, AR occluder is a generic model of a head representing user’s head in a 3D AR scene.

Usage:

Use OccluderPlugin to make a scene node be an occluder.
Use HeadTrackPlugin to make occluder’s node follow the head.

Models

Scene nodes can be attached to user’s head, they will follow its current pose (translation, orientation, and optionally scale). This attachment can be used for adding accessories such as headwear, or glasses, or other 3D models and effects.

Create a new empty project.
Import or build 3D models.
Import head or face model, to use them as a reference for alignment. You can leave the head model in the final scene and later use it as an occluder in the AR experience.

Import models

Create an empty node with identity transformation (position at the origin, no rotation and unit scale). This scene node will be attached to the head using HeadTrackPlugin. The plugin will continuously update transform (position, rotation and scale) of the node according to the current pose of a head. All children scene nodes will hierarchically include this transform. Pivot node can be seen as a virtual placeholder of a real head.

Create node

Make all 3D objects that will be attached to a head children of this node.

Set parent

Align 3D objects around the reference assigning relative transform respect to their parent pivot node. In AR they will be aligned with the real face/head exactly the same way. Relative transforms will be preserved while transform of parent node will follow the current head pose. In other words, a scene built around reference models will be renderer around user’s head and follow its pose (position, orientation and scale).

Align models

Remove reference objects (face model) from the scene. Optionally, leave the head model to later use it as an occluder utilizing the OccluderPlugin.
Export the final scene into .glb or embedded .gltf file.

Export gltf

Usage

Use HeadTrackPlugin to attach a scene node to the head. Children meshes will be attached to the head the same way they are positioned around reference models.
Use FaceTrackPlugin to attach a scene node to a face point. Children meshes will be attached to this point the same way they are positioned relative to the point of the reference face model.
Use OccluderPlugin to make scene node an occluder. Usually, occluder is a generic model of a head that is used to hide scene geometries behind real user’s head. In other words, it represents user’s head in the scene but not rendered itself (transparent).
Example for babylon.js
Example for three.js