BgReplacePlugin
Background replace plugin
@geenee/armature!ShaderPlugin replaces background region of an image. Segmentation mask defines image foreground that stays untouched. Foreground-background classification is based on two thresholds defining uncertainty interval. Probability above foreground threshold classifies pixel as foreground, below background threshold as background. FG pixels are kept untouched, BG pixels are replaced with corresponding pixels from the background texture. For pixels within uncertainty region weighted interpolation between image and background textures takes place. Weight is evaluated by scaling uncertainty interval and probability to [0..1]. Plugin depends on @geenee/bodyrenderers-common!MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize a @geenee/bodyrenderers-common!MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases accuracy of patching and reduces the size of the patch region. Other mask post-processing plugins may be used to fine-tune for particular use case.
Extends
Constructors
new BgReplacePlugin()
new BgReplacePlugin(
threshBg,threshFg,mirror):BgReplacePlugin
Constructor
Parameters
• threshBg: number = 0.4
Background threshold
• threshFg: number = 0.6
Foreground threshold
• mirror: boolean = false
Mirror background
Returns
Overrides
Properties
bgTexture?
protectedoptionalbgTexture:ImageTexture
Background texture
fragSrc?
protectedoptionalfragSrc:string
Code of fragment shader (copy shader by default)
Inherited from
inputs?
protectedoptionalinputs:string[]
Shader texture inputs (names of sampler uniforms)
Inherited from
loaded
loaded:
boolean
Loaded state
Inherited from
mirror
protectedmirror:boolean=false
Mirror background
ordinal
ordinal:
number
Ordinal number
Inherited from
renderer?
protectedoptionalrenderer:Renderer<SegmentationResult>
Renderer loaded the plugin
Inherited from
shader?
protectedoptionalshader:ShaderProgram
Rendering shader
Inherited from
shaderCtx?
protectedoptionalshaderCtx:WebGL2RenderingContext
Rendering context
Inherited from
size
protectedsize:Size
Image size
Inherited from
uniforms?
protectedoptionaluniforms:object
Shader uniforms as name-type map object
Index Signature
[key: string]: UniformType
Inherited from
vertSrc?
protectedoptionalvertSrc:string
Vertex shader source (copy shader by default)
Inherited from
Methods
background()
background():
undefined|ImageTexture
Texture replacing a background
Returns
undefined | ImageTexture
Background texture
dispose()
dispose():
void
Dispose video plugin
Returns
void
Inherited from
load()
load(
renderer):Promise<void>
Initialize plugin
Initializes resources required for shader effect. Allocates texture that will replace background.
Parameters
• renderer: Renderer<SegmentationResult>
Renderer this plugin is attached to
Returns
Promise<void>
Promise resolving when initialization is finished
Overrides
process()
process(
result,input):Promise<boolean>
Process the image
Applies background replacement with texture.
Parameters
• result: SegmentationResult
Results of video processing
• input: WebGLTexture
Current image texture
Returns
Promise<boolean>
True on success, false otherwise
Overrides
setMirror()
setMirror(
mirror):void
Set background mirror mode
Parameters
• mirror: boolean
Mirror flag
Returns
void
setupCamera()
setupCamera(
ratio,angle):void
Set camera parameters
Could be overridden to adjust plugin’s pipeline.
Parameters
• ratio: number
Aspect ration of input video
• angle: number
Vertical field of view in radians
Returns
void
Inherited from
setupVideo()
setupVideo(
size):void
Set video size
Adjusts shader and texture to a new size.
Parameters
• size: Size
Resolution of input video
Returns
void
Overrides
unload()
unload():
void
Reset plugin
Releases all resources and instances created in load().
Returns
void
Overrides
update()
update(
result,stream):Promise<void>
Sealed
Update the image
Main method implementing webgl shader effect or filter. ShaderRenderer keeps track of the current iamge texture that will be be rendered. ShaderPlugin uses the current texture as input and writes results to ShaderPlugin#output. ShaderPlugin#output becomes new current texture of ShaderRenderer. This way all ShaderPlugins attached to renderer organize a chain of effects applied on top of each other. Method process() implements shader effect itself, it’s intended to be overridden by effect authors.
Parameters
• result: SegmentationResult
Results of video processing
• stream: HTMLCanvasElement
Captured video frame
Returns
Promise<void>
Promise resolving when update is finished