@geenee/bodyrenderers-common

Renderer

@geenee/armature!Renderer is the core visualization and logical part of any application. It’s attached to the @geenee/armature!Engine. Basically, renders define two methods load() and update(). The first one is used to initialize assets and prepare the scene (lightning, environment map). The second is used to update the scene according to results of video processing. This’s where all the logic happens. Renderers can be extended with plugins. Plugins do simple rendering task, for example add object to that follows the head or render avatar overlay.

This package defines common utilities, types, and helpers used by renderers. Additionally, it contains universal plugins that provide features not related to final rendering. Ready-made renderers and plugins are provided by the following packages:

Mask Plugins

MaskUploadPlugin

Uploads segmentation mask to an image texture that can be reused by plugins next in the rendering pipeline. Uploading texture ones and reusing it speeds up rendering. Most plugins that post-process or utilize mask depend on MaskUploadPlugin and require it to be attached to renderer. Mask texture will be loaded and added to the corresponding Pose, Face, or Mask of the tracking results as maskTex field of BodyMaskTexture type. Evaluation of body segmentation mask must be enabled in @geenee/bodyprocessors!PoseProcessor#init setting @geenee/bodyprocessors!PoseParams#mask to true or @geenee/bodyprocessors!FaceProcessor#init setting @geenee/bodyprocessors!FaceParams#mask to true.

MaskUpscalePlugin

Performs smart upscaling of a segmentation mask on a pixel level using input image as a context provider. Increases the resolution of a segmentation mask preserving the consistency of foreground and background regions. BodyMaskHDPlugin is a processing (pre-render) stage plugin that can be attached to a @geenee/armature!ShaderRenderer. It depends on MaskUploadPlugin that must be attached to a renderer. High resolution mask will replace current maskTex within tracking results as described in MaskUploadPlugin. It can be utilized by plugins next in the rendering pipeline. Higher resolution segmentation increases quality and accuracy of partial patching and occluder components of the rendering.

MaskSmoothPlugin

Performs a smoothing convolution on a segmentation mask. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

MaskDilationPlugin

Performs the dilation operation on a segmentation mask. Dilation is a morphological operations adding pixels to the boundaries of masked objects in an image. The value of the output pixel is maximum value of all pixels in the neighbourhood which radius is a plugin’s parameter. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

MaskErosionPlugin

Performs the erosion operation on a segmentation mask. Erosion is a morphological operations removing pixels on the boundaries of masked objects in an image. The value of the output pixel is minimum value of all pixels in the neighbourhood which radius is a plugin’s parameter. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

MaskMorphPlugin

Performs a morphological operation on a segmentation mask. The most basic morphological ops are dilation and erosion. Dilation adds pixels to the boundaries of objects in an image, while erosion removes pixels on object boundaries. Value of the output pixel is maximum value of all pixels in the neighbourhood for dilation, and minimum for erosion. Radius of a square neighbourhood is a plugin’s parameter. Positive radius provides dilation while negative - erosion. Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases mask’s accuracy. Plugin may be combined with other mask post-processing plugins. Order in which operation on mask are applied is defined by order of plugins’ attachment.

Common Plugins

BodyPatchPlugin

Plugin patches (inpaints/erases) foreground region of image defined by body segmentation mask from @geenee/bodyprocessors!PoseProcessor. It may be used in avatar virtual try to remove parts of a user’s body that stick out (not covered). Plugin depends on MaskUploadPlugin that must be attached to the renderer to upload mask in an image texture. One may utilize MaskUpscalePlugin that provides higher resolution segmentation that significantly increases accuracy of patching and reduces size of patch region. Other mask post-processing plugins may be attached.

BgBlurPlugin

@geenee/armature!ShaderPlugin blurring background region of an image. Segmentation mask defines image foreground that stays untouched. Plugin depends on @geenee/bodyrenderers-common!MaskUploadPlugin that must be attached to the renderer to upload mask buffer in texture. One may utilize a @geenee/bodyrenderers-common!MaskUpscalePlugin providing higher resolution segmentation mask that significantly increases accuracy of patching and reduces the size of the patch region. Other mask post-processing plugins may be used to fine-tune for particular use case.

BrightnessPlugin

Plugin estimates parameters of lighting (brightness) observed in the current image and calls the provided callback with these parameters. This plugin can be used for automatic adjustment of intensities of lights on the scene, for example in callback user can scale intensity properties of lights and environment map according to estimated brightness. One can set output range, estimated value will not go below the minimum and [min..max] range will be scaled to [min..1.0]. Adaptive low pass smoothing is applied to the result.

BilateralPlugin

Plugin applying bilateral filter: non-linear, edge- preserving, and noise-reducing smoothing image filter.

Utility Plugins

CallbackPlugin

Simple plugin executing provided callback on processing results. Useful when, for example, application needs to show a message when no object is detected in camera’s field of view or pose is not optimal for virtual try on.

PoseFilterPlugin

Simple plugin filtering a list of detected poses by the provided predicate. Useful when application may need to ignore certain poses to provide the best quality of virtual try-on. For example to ensure that at least upper body of user is in the field of view.

FaceFilterPlugin

Simple plugin filtering a list of detected faces by the provided predicate. Useful when application may need to ignore certain head poses, for example when rotation angle is too big for virtual try-on.

PerfDevPlugin

Simple plugin measuring basic performance metrics of a renderer (fps). Executes provided callback when metrics are updated (every 30 rendered frames).