Skip to main content

Class: FaceMaskPlugin

@geenee/bodyrenderers-babylon.FaceMaskPlugin

Plugin assigning face mesh to geometry to a scene node

Plugin controls geometry of a scene node. Vertices are continuously updated using face mesh points estimated by @geenee/bodyprocessors!FaceProcessor. Node's mesh must have geometry compatible with detected face mesh, preferably created from the reference face model. The main requirement is that uv map must be compatible. The node can be seen as a virtual placeholder for real object. It's recommended to attach top-level nodes that don't have transforms relative to the root, otherwise this transforms will be applied to absolute positions of 3d points of the face (points will act as relative). One of approaches to create node for accurate face mask when modeling a scene is to import reference face model as top-level scene node and add one or more materials which textures are compatible with reference's uv map. And then instantiate FaceMaskPlugin for this scene node. In () plugin replaces geometry of attached node's mesh with one provided by FaceProcessor, defined indices, uv mapping and normals. As soon as uv maps of the scene mesh and the reference model are compatible all materials will be applied the same way as in the modeled scene. Vertices positions are updated in () method according to current face tracking estimations.

Hierarchy

Constructors

constructor

new FaceMaskPlugin(mesh?)

Constructor

Parameters

NameTypeDescription
mesh?MeshScene mesh to attach

Overrides

BabylonPlugin.constructor

Properties

loaded

loaded: boolean

Loaded state

Inherited from

BabylonPlugin.loaded


mesh

Protected Optional mesh: Mesh

Scene mesh to attach


ordinal

ordinal: number

Ordinal number

Inherited from

BabylonPlugin.ordinal


pointCont

Readonly pointCont: number = meshReference.length

Number of points in detected mesh


renderer

Protected Optional renderer: Renderer<FaceResult>

Renderer loaded the plugin

Inherited from

BabylonPlugin.renderer


scene

Protected Optional scene: Scene

Reference to a scene instance

Inherited from

BabylonPlugin.scene

Methods

dispose

dispose(): void

Dispose video plugin

Returns

void

Inherited from

BabylonPlugin.dispose


load

load(renderer): Promise<void>

Initialize plugin

FaceMaskPlugin replaces geometry of attached mesh by compatible with face estimated by FaceProcessor. As soon as uv maps of the node and the reference face model are compatible, materials/textures will be applied the same way as in the modeled 3d scene. Defines indices, uvs, normals, while positions are updated in ().

Parameters

NameTypeDescription
rendererRenderer<FaceResult>Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Overrides

BabylonPlugin.load


setMesh

setMesh(mesh?): Promise<void>

Set/attach a scene mesh

Rebuilds geometry of the mesh node to be compatible with face mesh points estimated by FaceProcessor. As soon as uv maps of the node and the reference face model are compatible, materials/textures will be applied the same way as in the modeled 3d scene.

Parameters

NameTypeDescription
mesh?MeshScene mesh node to attach

Returns

Promise<void>

Promise resolving when initialization is finished


setupCamera

setupCamera(ratio, angle): void

Set camera parameters

Could be overridden to adjust plugin's pipeline.

Parameters

NameTypeDescription
rationumberAspect ration of input video
anglenumberVertical field of view in radians

Returns

void

Inherited from

BabylonPlugin.setupCamera


setupVideo

setupVideo(size): void

Set video size

Could be overridden to adjust plugin's pipeline.

Parameters

NameTypeDescription
sizeSizeResolution of input video

Returns

void

Inherited from

BabylonPlugin.setupVideo


unload

unload(): void

Reset plugin

Releases all resources allocated in load(). Deletes cached reference to the scene object.

Returns

void

Inherited from

BabylonPlugin.unload


update

update(result, stream): Promise<void>

Update geometry of the mesh

Updates vertex positions according to points of the face estimated by @geenee/bodyprocessors!FaceProcessor. If face is not detected, plugin hides the attached mesh.

Parameters

NameTypeDescription
resultFaceResultResults of video processing
streamHTMLCanvasElementCaptured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

BabylonPlugin.update