Segmentation Masks
Segmentation masks associated with tracked objects (face, head,
body) now have the same BodyMask
type and added to an object as
mask
field when segmentation is enabled for a processor/engine.
Unification of mask type within tracked objects structure allows
plugins that use segmentation mask to be compatible with any types
of tracking. Most of mask plugins are now compatible with the
SegmentationResult
union type, so they can be used within face, body and hand based
virtual try-ons.
Pipeline of segmentation masks processing was completely restructured
to provide the best performance and more flexibility when using masks
in rendering or video processing. All plugins processing or utilizing
a mask now rely on the fact that it should be pre-uploaded to a texture.
MaskUploadPlugin
uploads segmentation mask to an image texture that can be reused by
plugins next in the rendering pipeline. Uploaded mask is added to the
corresponding tracked
Pose,
Face, or
Mask
as maskTex
field of type
BodyMaskTexture.
Next plugins update or use the current mask texture associated with a
tracked object. This allows to apply chains of plugins post-processing
the segmentation mask. For example, before doing background blur we
can apply erosion operation on the mask to remove a couple of pixels
on foreground boarders and smooth operator to remove sharp edges. This
is a quite powerful approach allowing to combine atomic operations on a
segmentation mask and fine-tune it for a particular use case or type of
virtual try-on. We introduce a set of plugins that process a mask.
Mask Post-Processing
We introduce a set of plugins for mask post-processing:
- MaskSmoothPlugin
performs blurring of a segmentation mask to smooth sharp
edges or propagate masked region outwards and inwards.
- MaskDilationPlugin
performs the dilation operation on a segmentation mask
adding pixels to the boundaries of masked objects.
- MaskErosionPlugin
performs the erosion operation on a segmentation mask
removing pixels on the boundaries of masked objects.
- MaskMorphPlugin
performs a morphological operation on a segmentation
mask combining dilation and erosion in one plugin.
Following the principle of plugin chaining, now plugins update and have access
to the same segmentation mask associated with a tracked object. This makes high
resolution versions of plugins obsolete. BodyPatchHDPlugin, BodypartPatchHDPlugin,
and OccluderMaskHDPlugin are removed from the Engeenee SDK as redundant.
BodyPatchPlugin,
BodypartPatchPlugin, and
OccluderMaskPlugin
become compatible with segmentation masks of any resolution uploaded to a texture.
While mask plugins automatically reconfigure themselves to the resolution
of a segmentation mask their constructors have mask size parameter. It is
recommended, though not required, to set it to the size that mask will have
at the stage where the plugin will be attached to a renderer. This will allow
to avoid unnecessary reconfiguration on runtime after renderer is initialized.
More information can be found here.
Mask upscaling was updated for the new pipeline and now works as other mask
plugins updating mask texture embedded into tracked objects by mask uploader.
BodyMaskHDPlugin
was renamed to MaskUpscalePlugin
.
Performance and accuracy of MaskUpscalePlugin was significantly improved.
Additionally, now it’s possible to choose number of x2 upscale levels
and thus resolution of output mask that can help to fine-tune between
performance and accuracy.
Temporal Mask Filter
Temporal filtering of segmentation masks provides for more stable results
with less noise and smoother changes of a mask over time. Yet temporal
filter doesn’t introduce any delay and reacts to changes immediately.
The main goal of temporal filtering is to resolve ambiguity in regions
that are hard to robustly classify as foreground. It’s available for all
processors that output segmentation masks, namely
PoseProcessor,
FaceProcessor,
and MaskProcessor.
We’ve developed blazingly fast implementation that has minimal impact
on performance and at the same time significantly increases accuracy.
Temporal mask filtering can be enabled by providing { smooth: true }
as mask
parameter of processor settings when calling Engine’s
init().
Resizable Segmentation
In this release we introduce resizable neural networks for body segmentation.
Resolution of evaluated segmentation masks can be adjusted, for a particular
use case one can choose between speed and accuracy. A lower resolution mask
can be evaluated much faster, while higher resolution provides higher density
of mask pixels for a region of interest. Smart upscaling of segmentation masks
allows to achieve good segmentation results even when mask size is reduced.
Another important improvement is that trackers adjust size of segmentation mask
according to aspect ratio of input video. When mask is evaluated for the whole
image, size of required padding is significantly reduced. This improves both,
accuracy of segmentation and density of mask pixels ratio to image pixels.
For face tracking it’s possible to choose the size of the segmentation mask by setting
maskSize parameter.
There are two modes of selecting region of segmentation in face tracker, first is to
do segmentation for the whole image that is useful for selfie-like use cases, second -
for an extended bounding box of the detected face, this option may work better when
face tracking is used in full-body experiences on AR mirrors. One may choose between
modes setting maskExt
processor parameter.
Mask Plugins
- BgBlurPlugin
blurs background region of an image.
- BodyPatchPlugin
patches (inpaints / erases) foreground region of an image.
- BodypartPatchPlugin
conditionally patches (inpaints / erases) foreground regions of
an image that are closer to virtual added objects than to the body.
All plugins that classify pixels as foreground or background are
parametrized by foreground threshold.
Other Changes
- Performance of BodypartPatchPlugin has been significantly improved.
- Adjustable radius of patching in BodyPatchPlugin and BodypartPatchPlugin.
- ShaderProgram supports changing uniforms on runtime using
setUniform().
Previously it might be done only in constructor of process().
- Support of vector uniforms.
- ShaderProgram may output a
grayscale image.
- Rename BlurMaskPlugin -> BgBlurPlugin.
- Optimization of WASM modules initialization.