@geenee/bodyrenderers-common

Renderer

@geenee/armature!Renderer is the core visualization and logical part of any application. It’s attached to the @geenee/armature!Engine. Basically, renders define two methods load() and update(). The first one is used to initialize assets and prepare the scene (lightning, environment map). The second is used to update the scene according to results of video processing. This’s where all the logic happens. Renderers can be extended with plugins. Plugins do simple rendering task, for example add object to that follows the head or render avatar overlay.

This package defines common utilities, types, and helpers used by renderers. Additionally, it contains universal plugins that provide features not related to final rendering. Ready-made renderers and plugins are provided by the following packages:

Mask Plugins

MaskUploadPlugin

Uploads segmentation mask to an image texture that can be reused by plugins next in the rendering pipeline. Uploading texture ones and reusing it speeds up rendering. Most plugins that post-process or utilize mask depend on MaskUploadPlugin and require it to be attached to renderer. Mask texture will be loaded and added to the corresponding Pose, Face, or Mask of the tracking results as maskTex field of BodyMaskTexture type. Evaluation of body segmentation mask must be enabled in @geenee/bodyprocessors!PoseProcessor#init setting @geenee/bodyprocessors!PoseParams#mask to true or @geenee/bodyprocessors!FaceProcessor#init setting @geenee/bodyprocessors!FaceParams#mask to true.

MaskUpscalePlugin

Performs smart upscaling of a segmentation mask on a pixel level using input image as a context provider. Increases the resolution of a segmentation mask preserving the consistency of foreground and background regions. BodyMaskHDPlugin is a processing (pre-render) stage plugin that can be attached to a @geenee/armature!ShaderRenderer. It depends on MaskUploadPlugin that must be attached to a renderer. High resolution mask will replace current maskTex within tracking results as described in MaskUploadPlugin. It can be utilized by plugins next in the rendering pipeline. Higher resolution segmentation increases quality and accuracy of partial patching and occluder components of the rendering.

MaskSmoothPlugin

Performs a smoothing convolution on a segmentation mask. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

MaskFilterPlugin

Performs temporal filtering of a segmentation mask that is based on minimization of entropy in probability space. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. It may be combined with other mask post-processing plugins. Order in which operation on mask are applied corresponds to the order in which plugins are attached to the renderer.

MaskSharpPlugin

Applies a smart sharpening filter on a segmentation mask. This plugin may significantly increase accuracy of a mask. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. It may be combined with other mask post-processing plugins. Order in which operation on mask are applied corresponds to the order in which plugins are attached to the renderer.

MaskDilationPlugin

Performs the dilation operation on a segmentation mask. Dilation is a morphological operations adding pixels to the boundaries of masked objects in an image. The value of the output pixel is maximum value of all pixels in the neighbourhood which radius is a plugin’s parameter. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

MaskErosionPlugin

Performs the erosion operation on a segmentation mask. Erosion is a morphological operations removing pixels on the boundaries of masked objects in an image. The value of the output pixel is minimum value of all pixels in the neighbourhood which radius is a plugin’s parameter. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

MaskMorphPlugin

Performs a morphological operation on a segmentation mask. The most basic morphological ops are dilation and erosion. Dilation adds pixels to the boundaries of objects in an image, while erosion removes pixels on object boundaries. Value of the output pixel is maximum value of all pixels in the neighbourhood for dilation, and minimum for erosion. Radius of a square neighbourhood is a plugin’s parameter. Positive radius provides dilation while negative - erosion. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

MaskBinaryPlugin

Applies a mask binarization agains provided threshold. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. It may be combined with other mask post-processing plugins. Order in which operation on mask are applied corresponds to the order in which plugins are attached to the renderer.

MaskStepPlugin

Performs mask binarization with smooth transition between 0 and 1 applying sigmoid function within threshold range. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. It may be combined with other mask post-processing plugins. Order in which operation on mask are applied corresponds to the order in which plugins are attached to the renderer.

Common Plugins

BodyPatchPlugin

Plugin patches (inpaints/erases) foreground region of image defined by body segmentation mask from @geenee/bodyprocessors!PoseProcessor. It may be used in avatar virtual try to remove parts of a user’s body that stick out (not covered). Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask in an image texture. One may utilize MaskUpscalePlugin that provides higher resolution segmentation that significantly increases accuracy of patching and reduces size of patch region. Other mask post-processing plugins may be attached.

BgBlurPlugin

@geenee/armature!ShaderPlugin blurring background region of an image. Segmentation mask defines image foreground that stays untouched. Plugin depends on @geenee/bodyrenderers-common!MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize a @geenee/bodyrenderers-common!MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases accuracy of patching and reduces the size of the patch region. Other mask post-processing plugins may be used to fine-tune for particular use case.

BgReplacePlugin

@geenee/armature!ShaderPlugin replaces background region of an image. Segmentation mask defines image foreground that stays untouched. Foreground-background classification is based on two thresholds defining uncertainty interval. Probability above foreground threshold classifies pixel as foreground, below background threshold as background. FG pixels are kept untouched, BG pixels are replaced with corresponding pixels from the background texture. For pixels within uncertainty region weighted interpolation between image and background textures takes place. Weight is evaluated by scaling uncertainty interval and probability to [0..1]. Plugin depends on @geenee/bodyrenderers-common!MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize a @geenee/bodyrenderers-common!MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases accuracy of patching and reduces the size of the patch region. Other mask post-processing plugins may be used to fine-tune for particular use case.

BrightnessPlugin

Plugin estimates parameters of lighting (brightness) observed in the current image and calls the provided callback with these parameters. This plugin can be used for automatic adjustment of intensities of lights on the scene, for example in callback user can scale intensity properties of lights and environment map according to estimated brightness. One can set output range, estimated value will not go below the minimum and [min..max] range will be scaled to [min..1.0]. Adaptive low pass smoothing is applied to the result.

BilateralPlugin

Plugin applying bilateral filter: non-linear, edge- preserving, and noise-reducing smoothing image filter.

VideoMergePlugin

@geenee/armature!ShaderPlugin merges input video stream with the background texture. Merge is done by linear interpolation between images and interpolation weight (alpha) is a tunable input parameter that can be changed on runtime for smooth transition between streams. Background texture is allocated by the plugin and has the same size as the input stream, it can be accessed and updated on runtime, this, for example, allows to replace input video stream with a background video.

Utility Plugins

CallbackPlugin

Simple plugin executing provided callback on processing results. Useful when, for example, application needs to show a message when no object is detected in camera’s field of view or pose is not optimal for virtual try on.

PoseFilterPlugin

Simple plugin filtering a list of detected poses by the provided predicate. Useful when application may need to ignore certain poses to provide the best quality of virtual try-on. For example to ensure that at least upper body of user is in the field of view.

FaceFilterPlugin

Simple plugin filtering a list of detected faces by the provided predicate. Useful when application may need to ignore certain head poses, for example when rotation angle is too big for virtual try-on.

PerfDevPlugin

Simple plugin measuring basic performance metrics of a renderer (fps). Executes provided callback when metrics are updated (every 30 rendered frames).

DelayPlugin

This is a factory function adding delay to a plugin it was called on. In delayed plugin update() method is called with detection results from one of the previous iterations of the tracking engine. This way we achieve effect of object following its controller with some delay. Delay may be used in combination with various twin plugins to achieve less artificial behaviour of twins that will repeat user movements without absolute synchronization in time and thus look more natural.