FaceMaskPlugin
Plugin assigning face mesh to geometry to a scene node
Plugin controls geometry of a scene node. Vertices are continuously updated using face mesh points estimated by @geenee/bodyprocessors!FaceProcessor. Node’s mesh must have geometry compatible with detected face mesh, preferably created from the reference face model. The main requirement is that uv map must be compatible. The node can be seen as a virtual placeholder for real object. It’s recommended to attach top-level nodes that don’t have transforms relative to the root, otherwise this transforms will be applied to absolute positions of 3d points of the face (points will act as relative). One of approaches to create node for accurate face mask when modeling a scene is to import reference face model as top-level scene node and add one or more materials which textures are compatible with reference’s uv map. And then instantiate FaceMaskPlugin for this scene node. In load() plugin replaces geometry of attached node’s mesh with one provided by FaceProcessor, defined indices, uv mapping and normals. As soon as uv maps of the scene mesh and the reference model are compatible all materials will be applied the same way as in the modeled scene. Vertices positions are updated in update() method according to current face tracking estimations.
Extends
Constructors
new FaceMaskPlugin()
new FaceMaskPlugin(
mesh?):FaceMaskPlugin
Constructor
Parameters
• mesh?: Mesh<BufferGeometry<NormalBufferAttributes>, Material | Material[], Object3DEventMap>
Scene mesh to attach
Returns
Overrides
Properties
loaded
loaded:
boolean
Loaded state
Inherited from
mesh?
protectedoptionalmesh:Mesh<BufferGeometry<NormalBufferAttributes>,Material|Material[],Object3DEventMap>
Scene mesh to attach
ordinal
ordinal:
number
Ordinal number
Inherited from
pointCont
readonlypointCont:number=meshReference.length
Number of points in detected mesh
renderer?
protectedoptionalrenderer:Renderer<FaceResult>
Renderer loaded the plugin
Inherited from
scene?
protectedoptionalscene:Scene
Reference to a scene instance
Inherited from
Methods
dispose()
dispose():
void
Dispose video plugin
Returns
void
Inherited from
load()
load(
renderer):Promise<void>
Initialize plugin
FaceMaskPlugin replaces geometry of attached mesh by compatible with face estimated by FaceProcessor. As soon as uv maps of the node and the reference face model are compatible, materials/textures will be applied the same way as in the modeled 3d scene. Defines indices, uvs, normals, while positions are updated in update().
Parameters
• renderer: Renderer<FaceResult>
Renderer this plugin is attached to
Returns
Promise<void>
Promise resolving when initialization is finished
Overrides
setMesh()
setMesh(
mesh?):Promise<void>
Set/attach a scene mesh
Rebuilds geometry of the mesh node to be compatible with face mesh points estimated by FaceProcessor. As soon as uv maps of the node and the reference face model are compatible, materials/textures will be applied the same way as in the modeled 3d scene.
Parameters
• mesh?: Mesh<BufferGeometry<NormalBufferAttributes>, Material | Material[], Object3DEventMap>
Scene mesh node to attach
Returns
Promise<void>
Promise resolving when initialization is finished
setupCamera()
setupCamera(
ratio,angle):void
Set camera parameters
Could be overridden to adjust plugin’s pipeline.
Parameters
• ratio: number
Aspect ration of input video
• angle: number
Vertical field of view in radians
Returns
void
Inherited from
setupVideo()
setupVideo(
size):void
Set video size
Could be overridden to adjust plugin’s pipeline.
Parameters
• size: Size
Resolution of input video
Returns
void
Inherited from
unload()
unload():
void
Reset plugin
Releases all resources allocated in load(). Deletes cached reference to the scene object.
Returns
void
Inherited from
update()
update(
result,stream):Promise<void>
Update geometry of the mesh
Updates vertex positions according to points of the face estimated by @geenee/bodyprocessors!FaceProcessor. If face is not detected, plugin hides the attached mesh.
Parameters
• result: FaceResult
Results of video processing
• stream: HTMLCanvasElement
Captured video frame
Returns
Promise<void>
Promise resolving when update is finished