OccluderMaskPlugin
Masked occluder plugin
Plugin constructs instances of OccluderMaskMaterial. Occluders are elements of a scene that are not rendered by themselves but still participate in occlusion queries. Usually, an occluder is a base mesh (average approximation) of a body representing its real counterpart in a scene. Occluders are used to mask visible virtual objects behind them (like geometries of a 3D scene behind user’s body). Masked occluder material is a more advanced version that takes into account a detected segmentation mask. It writes depth value only for pixels covered by a mask providing more realistic occlusion aligned with a body, additionally reducing effect of unnatural cuts in VTO. OccluderMaskMaterial is created by the plugin that owns and updates mask texture and sets corresponding uniforms of shaders (do not construct OccluderMaskMaterial directly). Plugin depends on @geenee/bodyrenderers-common!MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize a @geenee/bodyrenderers-common!MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases accuracy of patching and reduces the size of the patch region. Other mask post-processing plugins may be used to fine-tune for particular use case.
Extends
Section titled “Extends”Constructors
Section titled “Constructors”Constructor
Section titled “Constructor”new OccluderMaskPlugin(
thresh):OccluderMaskPlugin
Constructor
Parameters
Section titled “Parameters”thresh
Section titled “thresh”number = 0.55
Foreground threshold
Returns
Section titled “Returns”OccluderMaskPlugin
Overrides
Section titled “Overrides”Properties
Section titled “Properties”loaded
Section titled “loaded”loaded:
boolean
Loaded state
Inherited from
Section titled “Inherited from”maskRect
Section titled “maskRect”
protectedmaskRect:Vector4
Rect region of the segmentation mask
maskTexture?
Section titled “maskTexture?”
protectedoptionalmaskTexture:BaseTexture
Texture storing the segmentation mask
ordinal
Section titled “ordinal”ordinal:
number
Ordinal number
Inherited from
Section titled “Inherited from”renderer?
Section titled “renderer?”
protectedoptionalrenderer:Renderer<PoseResult|FaceResult>
Renderer loaded the plugin
Inherited from
Section titled “Inherited from”scene?
Section titled “scene?”
protectedoptionalscene:Scene
Reference to a scene instance
Inherited from
Section titled “Inherited from”thresh
Section titled “thresh”
protectedthresh:number=0.55
Foreground threshold
Methods
Section titled “Methods”createMaterial()
Section titled “createMaterial()”createMaterial():
null|OccluderMaskMaterial
Create OccluderMaskMaterial
Occluders are elements of a scene that are not rendered by themselves but still participate in occlusion queries. Usually, occluder is a base mesh (average approximation) of a body representing its real counterpart in a scene. Occluders are used to mask visible virtual objects behind them (like geometries of a 3D scene behind user’s body). Masked occluder material is a more advanced version that takes into account a detected hd body segmentation mask. High resolution mask significantly increases accuracy. It writes depth value only for pixels covered by a mask providing more realistic occlusion aligned with a body, additionally reducing effect of unnatural cuts in VTO. OccluderMaskMaterial is created by the plugin, that owns and updates the mask texture and sets shader uniforms. (please do not construct OccluderMaskMaterial directly).
Returns
Section titled “Returns”null | OccluderMaskMaterial
Instance of the masked occluder material
dispose()
Section titled “dispose()”dispose():
void
Dispose video plugin
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”load()
Section titled “load()”load(
renderer):Promise<void>
Initialize plugin
Allocates a texture for the segmentation mask.
Parameters
Section titled “Parameters”renderer
Section titled “renderer”Renderer<PoseResult | FaceResult>
Renderer this plugin is attached to
Returns
Section titled “Returns”Promise<void>
Promise resolving when initialization is finished
Overrides
Section titled “Overrides”setupCamera()
Section titled “setupCamera()”setupCamera(
ratio,angle):void
Set camera parameters
Could be overridden to adjust plugin’s pipeline.
Parameters
Section titled “Parameters”number
Aspect ration of input video
number
Vertical field of view in radians
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”setupVideo()
Section titled “setupVideo()”setupVideo(
size):void
Set video size
Could be overridden to adjust plugin’s pipeline.
Parameters
Section titled “Parameters”Resolution of input video
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”unload()
Section titled “unload()”unload():
void
Reset plugin
Releases all resources allocated in load(). Deletes the segmentation mask texture.
Returns
Section titled “Returns”void
Overrides
Section titled “Overrides”update()
Section titled “update()”update(
result,stream):Promise<void>
Update segmentation mask
Updates internal segmentation mask texture and caches its box region uniform. Mask texture and uniform are provided to every occluder material created by the plugin before binding a mesh.
Parameters
Section titled “Parameters”result
Section titled “result”Pose estimation results
stream
Section titled “stream”HTMLCanvasElement
Captured video frame
Returns
Section titled “Returns”Promise<void>
Promise resolving when update is finished