Class: OccluderMaskPlugin

@geenee/bodyrenderers-babylon.OccluderMaskPlugin

Masked occluder plugin

Plugin constructs instances of OccluderMaskMaterial. Occluders are elements of a scene that are not rendered by themselves but still participate in occlusion queries. Usually, an occluder is a base mesh (average approximation) of a body representing its real counterpart in a scene. Occluders are used to mask visible virtual objects behind them (like geometries of a 3D scene behind user's body). Masked occluder material is a more advanced version that takes into account a detected segmentation mask. It writes depth value only for pixels covered by a mask providing more realistic occlusion aligned with a body, additionally reducing effect of unnatural cuts in VTO. OccluderMaskMaterial is created by the plugin that owns and updates mask texture and sets corresponding uniforms of shaders (do not construct OccluderMaskMaterial directly). Plugin depends on @geenee/bodyrenderers-common!MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize a @geenee/bodyrenderers-common!MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases accuracy of patching and reduces the size of the patch region. Other mask post-processing plugins may be used to fine-tune for particular use case.

Hierarchy

• BabylonPlugin<PoseResult | FaceResult>

  ↳ OccluderMaskPlugin

Constructors

constructor

• new OccluderMaskPlugin()

Constructor

Inherited from

BabylonPlugin.constructor

Properties

loaded

• loaded: boolean

Loaded state

Inherited from

BabylonPlugin.loaded

---

maskRect

• Protected maskRect: Vector4

Rect region of the segmentation mask

---

maskTexture

• Protected Optional maskTexture: BaseTexture

Texture storing the segmentation mask

---

ordinal

• ordinal: number

Ordinal number

Inherited from

BabylonPlugin.ordinal

---

renderer

• Protected Optional renderer: Renderer<PoseResult | FaceResult>

Renderer loaded the plugin

Inherited from

BabylonPlugin.renderer

---

scene

• Protected Optional scene: Scene

Reference to a scene instance

Inherited from

BabylonPlugin.scene

Methods

createMaterial

▸ createMaterial(): null | OccluderMaskMaterial

Create OccluderMaskMaterial

Occluders are elements of a scene that are not rendered by themselves but still participate in occlusion queries. Usually, occluder is a base mesh (average approximation) of a body representing its real counterpart in a scene. Occluders are used to mask visible virtual objects behind them (like geometries of a 3D scene behind user's body). Masked occluder material is a more advanced version that takes into account a detected hd body segmentation mask. High resolution mask significantly increases accuracy. It writes depth value only for pixels covered by a mask providing more realistic occlusion aligned with a body, additionally reducing effect of unnatural cuts in VTO. OccluderMaskMaterial is created by the plugin, that owns and updates the mask texture and sets shader uniforms. (please do not construct OccluderMaskMaterial directly).

Returns

null | OccluderMaskMaterial

Instance of the masked occluder material

---

dispose

▸ dispose(): void

Dispose video plugin

Returns

void

Inherited from

BabylonPlugin.dispose

---

load

▸ load(renderer): Promise<void>

Initialize plugin

Allocates a texture for the segmentation mask.

Parameters

Name	Type	Description
renderer	Renderer<PoseResult | FaceResult>	Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Overrides

BabylonPlugin.load

---

setupCamera

▸ setupCamera(ratio, angle): void

Set camera parameters

Could be overridden to adjust plugin's pipeline.

Parameters

Name	Type	Description
ratio	number	Aspect ration of input video
angle	number	Vertical field of view in radians

Returns

void

Inherited from

BabylonPlugin.setupCamera

---

setupVideo

▸ setupVideo(size): void

Set video size

Could be overridden to adjust plugin's pipeline.

Parameters

Name	Type	Description
size	Size	Resolution of input video

Returns

void

Inherited from

BabylonPlugin.setupVideo

---

unload

▸ unload(): void

Reset plugin

Releases all resources allocated in load(). Deletes the segmentation mask texture.

Returns

void

Overrides

BabylonPlugin.unload

---

update

▸ update(result, stream): Promise<void>

Update segmentation mask

Updates internal segmentation mask texture and caches its box region uniform. Mask texture and uniform are provided to every occluder material created by the plugin before binding a mesh.

Parameters

Name	Type	Description
result	PoseResult | FaceResult	Pose estimation results
stream	HTMLCanvasElement	Captured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

BabylonPlugin.update