Segmentation Masks
Segmentation masks associated with tracked objects (face, head,
body) now have the same BodyMask type and added to an object as
mask field when segmentation is enabled for a processor/engine.
Unification of mask type within tracked objects structure allows
plugins that use segmentation mask to be compatible with any types
of tracking. Most of mask plugins are now compatible with the
SegmentationResult
union type, so they can be used within face, body and hand based
virtual try-ons.
Pipeline of segmentation masks processing was completely restructured
to provide the best performance and more flexibility when using masks
in rendering or video processing. All plugins processing or utilizing
a mask now rely on the fact that it should be pre-uploaded to a texture.
MaskUploadPlugin
uploads segmentation mask to an image texture that can be reused by
plugins next in the rendering pipeline. Uploaded mask is added to the
corresponding tracked
Pose,
Face, or
Mask
as maskTex field of type
BodyMaskTexture.
Next plugins update or use the current mask texture associated with a tracked object. This allows to apply chains of plugins post-processing the segmentation mask. For example, before doing background blur we can apply erosion operation on the mask to remove a couple of pixels on foreground boarders and smooth operator to remove sharp edges. This is a quite powerful approach allowing to combine atomic operations on a segmentation mask and fine-tune it for a particular use case or type of virtual try-on. We introduce a set of plugins that process a mask.
Mask Post-Processing
We introduce a set of plugins for mask post-processing:
- MaskSmoothPlugin performs blurring of a segmentation mask to smooth sharp edges or propagate masked region outwards and inwards.
- MaskDilationPlugin performs the dilation operation on a segmentation mask adding pixels to the boundaries of masked objects.
- MaskErosionPlugin performs the erosion operation on a segmentation mask removing pixels on the boundaries of masked objects.
- MaskMorphPlugin performs a morphological operation on a segmentation mask combining dilation and erosion in one plugin.
Following the principle of plugin chaining, now plugins update and have access to the same segmentation mask associated with a tracked object. This makes high resolution versions of plugins obsolete. BodyPatchHDPlugin, BodypartPatchHDPlugin, and OccluderMaskHDPlugin are removed from the Engeenee SDK as redundant. BodyPatchPlugin, BodypartPatchPlugin, and OccluderMaskPlugin become compatible with segmentation masks of any resolution uploaded to a texture.
While mask plugins automatically reconfigure themselves to the resolution of a segmentation mask their constructors have mask size parameter. It is recommended, though not required, to set it to the size that mask will have at the stage where the plugin will be attached to a renderer. This will allow to avoid unnecessary reconfiguration on runtime after renderer is initialized.
More information can be found here.
Mask Upscaler
Mask upscaling was updated for the new pipeline and now works as other mask
plugins updating mask texture embedded into tracked objects by mask uploader.
BodyMaskHDPlugin was renamed to MaskUpscalePlugin.
Performance and accuracy of MaskUpscalePlugin was significantly improved. Additionally, now it's possible to choose number of x2 upscale levels and thus resolution of output mask that can help to fine-tune between performance and accuracy.
Temporal Mask Filter
Temporal filtering of segmentation masks provides for more stable results
with less noise and smoother changes of a mask over time. Yet temporal
filter doesn't introduce any delay and reacts to changes immediately.
The main goal of temporal filtering is to resolve ambiguity in regions
that are hard to robustly classify as foreground. It's available for all
processors that output segmentation masks, namely
PoseProcessor,
FaceProcessor,
and MaskProcessor.
We've developed blazingly fast implementation that has minimal impact
on performance and at the same time significantly increases accuracy.
Temporal mask filtering can be enabled by providing { smooth: true }
as mask
parameter of processor settings when calling Engine's
init().
Resizable Segmentation
In this release we introduce resizable neural networks for body segmentation. Resolution of evaluated segmentation masks can be adjusted, for a particular use case one can choose between speed and accuracy. A lower resolution mask can be evaluated much faster, while higher resolution provides higher density of mask pixels for a region of interest. Smart upscaling of segmentation masks allows to achieve good segmentation results even when mask size is reduced.
Another important improvement is that trackers adjust size of segmentation mask according to aspect ratio of input video. When mask is evaluated for the whole image, size of required padding is significantly reduced. This improves both, accuracy of segmentation and density of mask pixels ratio to image pixels.
For face tracking it's possible to choose the size of the segmentation mask by setting maskSize parameter. There are two modes of selecting region of segmentation in face tracker, first is to do segmentation for the whole image that is useful for selfie-like use cases, second - for an extended bounding box of the detected face, this option may work better when face tracking is used in full-body experiences on AR mirrors. One may choose between modes setting maskExt processor parameter.
Mask Plugins
- BgBlurPlugin blurs background region of an image.
- BodyPatchPlugin patches (inpaints / erases) foreground region of an image.
- BodypartPatchPlugin conditionally patches (inpaints / erases) foreground regions of an image that are closer to virtual added objects than to the body.
All plugins that classify pixels as foreground or background are parametrized by foreground threshold.
Other Changes
- Performance of BodypartPatchPlugin has been significantly improved.
- Adjustable radius of patching in BodyPatchPlugin and BodypartPatchPlugin.
- ShaderProgram supports changing uniforms on runtime using setUniform(). Previously it might be done only in constructor of process().
- Support of vector uniforms.
- ShaderProgram may output a grayscale image.
- Rename BlurMaskPlugin -> BgBlurPlugin.
- Optimization of WASM modules initialization.