Skip to main content

Face Tracking Scenes

Occluders​

Occluders are visual elements that represent real objects in a scene and are used to hide parts of AR objects behind them. They are not rendered themselves but participate in depth test, in other words they only signal not to draw parts of meshes behind them being completely transparent at the same time. For example in face tracking, AR occluder is a generic model of a head representing user’s head in a 3D AR scene.

Usage:

Models​

In this section we will explain how to prepare models and build scenes for face tracking applications. Face tracking estimates face mesh and head pose. This allows to attach 3D models to the head or to the particular face point. The later can provide better alignment for models like glasses or beards so model will follow selected keypoint and not head in general.

Basics​

General approach on preparing scenes for face tracking is to align meshes relative to reference head or face model. And attach corresponding scene node to head or face point using one of face tracking plugins. Then mesh will follow user's head/face and will be aligned the same way it was aligned relative to a reference model.

In general, preparation of models for face tracking is the same as any other 3D model and you can use any 3D tool you prefer: Blender, Maya, Cinema4D, etc. The main difference is that in the final scene model should be aligned with the reference head or face. In the next sections we will describe how to do it.

Models Attached to a Head​

Scene nodes can be attached to user's head, they will follow its current pose (translation, orientation, and optionally scale). This attachment can be used for adding accessories such as headwear, or glasses, or other 3D models and effects.

How to prepare models attached to a head.

Models Attached to a Face Point​

Scene nodes can be attached to points on the face, they will follow these points plus rotation and scale of the head. This attachment can be used for adding accessories like glasses, mustaches or other 3D models and effects.

How to prepare models attached to a face point.

Usage​

  • Use HeadTrackPlugin to attach a scene node to the head. Children meshes will be attached to the head the same way they are positioned around reference models.
  • Use FaceTrackPlugin to attach a scene node to a face point. Children meshes will be attached to this point the same way they are positioned relative to the point of the reference face model.
  • Use OccluderPlugin to make scene node an occluder. Usually, occluder is a generic model of a head that is used to hide scene geometries behind real user's head. In other words, it represents user’s head in the scene but not rendered itself (transparent).
  • Example for babylon.js
  • Example for three.js