PatchPartPlugin
Patch part plugin
Plugin conditionally patches (inpaints / erases) foreground regions of an input image. Foreground is defined by the body segmentation mask provided by a PoseProcessor or by a FaceProcessor. There are 2 types of regions: “patch” and “keep”, both are defined by corresponding sets of scene meshes. Plugin patches foreground/masked pixels that belong to “patch” regions but are not part of “keep” regions. This can be used in apparel virtual try-on to remove parts of a body that stick out of (not covered by) an outfit. In this case, “patch” region is defined by outfit meshes and “keep” region is the reference body model that at the same time serves as occluder. The plugin may optionally update segmentation mask and discard foreground regions that were patched, in other words flag patched pixels as background. The PatchPartPlugin is compatible with renderers derived from the BabylonUniRenderer. Plugin depends on @geenee/bodyrenderers-common!MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize a @geenee/bodyrenderers-common!MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases accuracy of patching and reduces the size of the patch region. Other mask post-processing plugins may be used to fine-tune for particular use case.
Extends
Section titled “Extends”Constructors
Section titled “Constructors”Constructor
Section titled “Constructor”new PatchPartPlugin(
thresh,radius,updateMask,maskSize):PatchPartPlugin
Constructor
Parameters
Section titled “Parameters”thresh
Section titled “thresh”number = 0.55
Foreground threshold
radius
Section titled “radius”number = 512
Radius of foreground search
updateMask
Section titled “updateMask”boolean = false
Update segmentation mask
maskSize
Section titled “maskSize”Size = ...
Size of segmentation mask
Returns
Section titled “Returns”PatchPartPlugin
Overrides
Section titled “Overrides”Properties
Section titled “Properties”fragSrc?
Section titled “fragSrc?”
protectedoptionalfragSrc:string
Code of fragment shader (copy shader by default)
Inherited from
Section titled “Inherited from”inputs?
Section titled “inputs?”
protectedoptionalinputs:string[]
Shader texture inputs (names of sampler uniforms)
Inherited from
Section titled “Inherited from”loaded
Section titled “loaded”loaded:
boolean
Loaded state
Inherited from
Section titled “Inherited from”maskSize
Section titled “maskSize”
protectedmaskSize:Size
Size of segmentation mask
ordinal
Section titled “ordinal”ordinal:
number
Ordinal number
Inherited from
Section titled “Inherited from”radius
Section titled “radius”
protectedradius:number=512
Radius of foreground search
renderer?
Section titled “renderer?”
protectedoptionalrenderer:Renderer<SegmentationResult>
Renderer loaded the plugin
Inherited from
Section titled “Inherited from”shader?
Section titled “shader?”
protectedoptionalshader:ShaderProgram
Rendering shader
Inherited from
Section titled “Inherited from”shaderCtx?
Section titled “shaderCtx?”
protectedoptionalshaderCtx:WebGL2RenderingContext
Rendering context
Inherited from
Section titled “Inherited from”
protectedsize:Size
Image size
Inherited from
Section titled “Inherited from”thresh
Section titled “thresh”
protectedthresh:number=0.55
Foreground threshold
uniforms?
Section titled “uniforms?”
protectedoptionaluniforms:object
Shader uniforms as name-type map object
Index Signature
Section titled “Index Signature”[key: string]: UniformType
Inherited from
Section titled “Inherited from”updateMask
Section titled “updateMask”
protectedupdateMask:boolean=false
Update segmentation mask
vertSrc?
Section titled “vertSrc?”
protectedoptionalvertSrc:string
Vertex shader source (copy shader by default)
Inherited from
Section titled “Inherited from”Methods
Section titled “Methods”dispose()
Section titled “dispose()”dispose():
void
Dispose video plugin
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”load()
Section titled “load()”load(
renderer):Promise<void>
Initialize plugin
Initializes resources required for conditional body patching.
Parameters
Section titled “Parameters”renderer
Section titled “renderer”Renderer this plugin is attached to
Returns
Section titled “Returns”Promise<void>
Promise resolving when initialization is finished
Overrides
Section titled “Overrides”process()
Section titled “process()”process(
result,input):Promise<boolean>
Process the image
Patches (inpaints) foreground image pixels according to the provided segmentation mask. Patched pixels must also belong to “patch” region and not be a part of “keep” region. Regions are defined by corresponding sets of scene meshes.
Parameters
Section titled “Parameters”result
Section titled “result”Results of video processing
WebGLTexture
Current image texture
Returns
Section titled “Returns”Promise<boolean>
True on success, false otherwise
Overrides
Section titled “Overrides”renderParts()
Section titled “renderParts()”
protectedrenderParts():null|WebGLTexture
Render patch and keep meshes
Rendered meshes provide mask of patch/keep regions.
Returns
Section titled “Returns”null | WebGLTexture
Mask texture
setParts()
Section titled “setParts()”setParts(
patchParts,keepParts):void
Set meshes defining patch regions
Parameters
Section titled “Parameters”patchParts
Section titled “patchParts”AbstractMesh[] = []
Meshes defining “patch” regions
keepParts
Section titled “keepParts”AbstractMesh[] = []
Meshes defining “keep” regions
Returns
Section titled “Returns”void
setupCamera()
Section titled “setupCamera()”setupCamera(
ratio,angle):void
Set camera parameters
Could be overridden to adjust plugin’s pipeline.
Parameters
Section titled “Parameters”number
Aspect ration of input video
number
Vertical field of view in radians
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”setupVideo()
Section titled “setupVideo()”setupVideo(
size):void
Set video size
Adjusts shader and texture to a new size.
Parameters
Section titled “Parameters”Resolution of input video
Returns
Section titled “Returns”void
Overrides
Section titled “Overrides”unload()
Section titled “unload()”unload():
void
Reset plugin
Releases all resources and instances created in load().
Returns
Section titled “Returns”void
Overrides
Section titled “Overrides”update()
Section titled “update()”update(
result,stream):Promise<void>
Sealed
Update the image
Main method implementing webgl shader effect or filter. ShaderRenderer keeps track of the current iamge texture that will be be rendered. ShaderPlugin uses the current texture as input and writes results to ShaderPlugin#output. ShaderPlugin#output becomes new current texture of ShaderRenderer. This way all ShaderPlugins attached to renderer organize a chain of effects applied on top of each other. Method process() implements shader effect itself, it’s intended to be overridden by effect authors.
Parameters
Section titled “Parameters”result
Section titled “result”Results of video processing
stream
Section titled “stream”HTMLCanvasElement
Captured video frame
Returns
Section titled “Returns”Promise<void>
Promise resolving when update is finished