Release v0.7.5
Hair Segmentation
The most important new feature added in this release is Hair Segmentation. We have added the new type of tracking that allows to implement virtual try-on effects like hair recoloring, patching of hairs in headwear virtual try-on, and many more. Hair Segmentation is GPGPU powered and real-time, it supports smooth visual effects on mobile devices.
The standard set of core tools to enable hair segmentation in an app:
Already available plugins implementing processings of segmentation masks are compatible with hair segmentation mask by default. Developers can use familiar building blocks to assemble pipelines of visually rich AR effects. ShaderPlugin can be used as a starting point of a custom visual effect on top of hair segmentation mask.
Mask Post-Processing
MaskBinaryPlugin applying binarization operation to a segmentation mask. This plugin can be used within mask processing pipeline to filter pixels based on their probability and separate foreground and background pixels by a threshold.
MaskStepPlugin
is a similar but more advanced version of the plugin applying smoothstep
operation between two thresholds. Compared to more agressive Heaviside
step function used in simple binarization, smoothstep provides smooth
transition of mask value between foreground and background regions still
separating them. We recommend using MaskStepPlugin for smoother visuals.
MaskFilterPlugin performs temporal filtering of a segmentation mask that is based on minimization of entropy in probability space. It can be integrated into a mask post-processing pipeline to decrease temporal noise.
One notable new plugin designed specifically for hair segmentation mask is
MaskSharpPlugin.
When used in post-processing pipeline of a segmentation mask it significantly
improves its accuracy. For example, in case of hair segmentation it can even
highlight separate locks of hairs that were missed by a neural network. We
recommend to apply a slight spacial smoothing (about 0.5) of a mask before
MaskSharpPlugin. If not provided by the Processor temporal smoothing of
a segmentation mask can also improve final results. While improvement of
mask quality is significant MaskSharpPlugin is really fast and can be used
in mobile browsers without sacrificing frames per second.
Body Patching
In this release we reimplemented body patching processing and corresponding plugins. Partial body patching is now not limited in radius of search for the closes body part defined by “patch” and “keep” meshes. This means that no matter how long is the distance between a pixel of a segmentation mask and the closest mesh it’ll be correctly classified as “patch” or “keep” pixel. Moreover this stage of processing is now blazingly fast and processing time doesn’t depend on a search radius. This makes usage of body patching more convenient across all virtual try-on types. Additionally, the same algorithm is applied within patching procedure itself improving it’s performance, quality, and accuracy. The same improvements and changes are applied within full body patching.
On top of that, to improve quality of image processing and overcome floating point precision limits in shaders on mobile devices we implement multi-stage refinement of results of intermediate processing stages. Additionally, memory access pattern is optimized for all GPGPU kernels across Engeenee SDK.
Wrist Tracking
Hand pose detection was optimized to provide more precise and smooth tracking. We’ve implemented totally new approach for wrist detection that provides much more robust and stable results and overcomes major disadvantages of the previous algorithm. New wrist estimation borrows many principles from phalanx refinement procedure introduced in the previous release and bringing promising results. Wrist is approximated by an elliptical cylindroid with subpixel accuracy. New detector has much better performance, it’s at least twice faster, thus virtual try-on of watches and bracelets has gained significant boost in FPS. On top of that, GPGPU kernels of wrist detector were significantly optimized for real-time usage on mobile devices. Wrist detection can be enabled setting corresponding parameter. WristTrackPlugin may be utilized to align scene node with a wrist estimation.
HandAlignPlugin has been considerably reworked. Now we utilize minimization of back-projection error when fitting hand armature in detected keypoints and iteratively converge skeleton in approximated pose to keypoints at the same time preserving relative lengths and scales. We utilized similar approach when aligning a body armature. HandAlignPlugin still requires some work to be 100% ready for virtual try-on of gloves, we are planning to complete implementation within the next release cycle.
Pose Alignment
This release introduces number of improvements in alignment of armature and detected 3D pose including more natural kinematics, both forward and inverse. We improved fitting of a skeleton into detected points relaxing some hard requirements that when enforced lead to less natural deformations of a mesh. Kinematics of shoulder bones has been reworked to provide less deformations within shoulders and underarms areas. We also fixed rescaling issue near shoulders using the same approach we implemented for limbs in the previous version of the SDK.
Spine curve
parameter of ClothAlignPlugin has been fixed and when not specified or
set to null | undefined default 1.0 value is used. At the same time
0.0 value is now treated properly and disables curvature recovery.
Behaviour of spine bones is also improved to reduce unnatural rescaling.
Image Texture
ImageTexture supports generation of mipmap levels via new convenient API. Mipmaps can considerably improve performance of certain image processing algorithms or visual effect shaders. Number of mipmap levels required to reach 1x1 resolution is evaluated internally. Mipmaps are generated automatically on texture update or upload. ShaderProgram provides an option to opt-in for output texture with auto-generated mipmaps that can be useful when chaining image processing stages.
New read() method
of a ShaderProgram copies GPU texture data to the host memory providing
access to raw pixel values.
Helper TextureReader class
provides functionality to get texture data on the client side and use
it within image processing pipelines. The important utility provided by
TextureReader is non-blocking asynchronous read of texture data using
readAsync().
In scenarios where data is not needed immediately or can be accessed with
a delay, usage of readAsync() can considerably improve performance as
blocking reads stall the whole GPU pipeline and wait until all operations
having the texture as a dependency are finished, this in turn significantly
decreases performance and utilization of a GPU device. Implementation of
readAsync() is thread-safe and correctly handles WebGL state recovery.
Generation of mipmaps and asynchronous read of textures allowed to reimplement BrightnessPlugin and derived LightsPlugin and make them very fast. Now they can be used in rendering pipelines without sacrificing frames per second. Additionally BrightnessPlugin was fine-tuned to react on illumination changes mush faster.
API Improvements
We’ve improved API of video capture setup and made it more flexible. Previously,
we had two sets of capture options available. One option is to provide standard
MediaStreamConstraints defining set of device capabilities and allowing complete
control over selection of a video device. The second option is our simplified
VideoParams that provide basic
selection of a video device. The main downside was that advanced features like
rotation or cropping of the input video streams were available only through
VideoParams. To effectively resolve this limitation we’ve merged VideoParams
and MediaStreamConstraints options. Now fine-grained specification of the
device is available via opts
field of VideoParams. This allows for example to request specific deviceId
or even focus or white balance modes. Provided MediaStreamConstraints have
higher priority than the rest or parameters of VideoParams.
SDK users can provide custom options to underlying babylon.js or three.js renderer when initializing BabylonUniRenderer or ThreeRenderer. This might be useful when more fine-grained control over rendering pipeline or canvas settings are required.
Other Changes
- Documentation page of Engeenee SDK is now generated by Starlight/Astro.
- Several fixes where made in saving state of rendering pipeline when WebGL context is shared between several clients.
- User can provide custom options to underlying babylon.js or three.js renderer when constructing BabylonUniRenderer or ThreeRenderer.
- Fine-tuning of temporal filter for 2D and 3D pose keypoints.
- Optimization of memory access in GPGPU kernels with performance improvement.