Point Tracking Scenes

Face Tracking Scenes

Occluders

Occluders are visual elements that represent real objects in a scene and are used to hide parts of AR objects behind them. They are not rendered themselves but participate in depth test, in other words they only signal not to draw parts of meshes behind them being completely transparent at the same time. For example in face tracking, AR occluder is a generic model of a head representing user’s head in a 3D AR scene.

Usage:

Use OccluderPlugin to make a scene node be an occluder.
Use HeadTrackPlugin to make occluder’s node follow the head.

Scene

Create a scene (Blender):

Create a new empty project.
Import or build 3D models.
Import face model to use it as a reference for alignment. Optionally import head model to later use it as an occluder in the AR experience.

Import models

Create an empty node. This scene node will be attached to the face point using FaceTrackPlugin. Plugin will continuously update transform (translation, rotation and scale) of the node according to the current estimation of the face mesh. Children nodes will hierarchically include this transform. This pivot node can be seen as a virtual placeholder of a real point on the face.

Create node

Set initial translation of the parent/pivot node equal to coordinates of the face point. This will help to setup relative transforms of children meshes that properly align them with the face.
Select a vertex of the face model in Edit Mode you would like 3D object to be attached to.

Face vertex

Snap cursor to the selected vertex. Right Click -> Snap Vertices -> Cursor to Selected.

Snap cursor

Snap pivot node’s position to the cursor. In Oject Mode select pivot scene node and Right Click -> Snap -> Selection to Cursor.

Snap pivot

Make all 3D objects that will be attached to the selected face point children of the pivot node.

Set parent

Align 3D objects with the reference face model assigning relative transforms respect to their parent pivot node. In AR they will be aligned with the real face exactly the same way. Relative transforms will be preserved while transform of parent node will follow the pose of the face point. In other words, a scene built around reference model will be renderer around user’s head and follow the selected face point.

Align models

Remove reference face model from the scene.

Remove face

Export the final scene into .glb or embedded .gltf file.

Export gltf

Find index of the face point in face map.
Use FaceTrackPlugin to attach the pivot node to the face point.

Usage

Use HeadTrackPlugin to attach a scene node to the head. Children meshes will be attached to the head the same way they are positioned around reference models.
Use FaceTrackPlugin to attach a scene node to a face point. Children meshes will be attached to this point the same way they are positioned relative to the point of the reference face model.
Use OccluderPlugin to make scene node an occluder. Usually, occluder is a generic model of a head that is used to hide scene geometries behind real user’s head. In other words, it represents user’s head in the scene but not rendered itself (transparent).
Example for babylon.js
Example for three.js