Skip to content
Engeenee
Blog

Release v0.7.3

Evgeny Peshkov
Evgeny Peshkov
CTO @ Geenee

Hand Detection

This release makes a big step in hand detection and tracking adding many tools and plugins. Hand tracking can be used to the full extent including alignment of hand armature, more advanced and precise model of a hand for rings virtual try-on, improvement of wrist tracking for watches and bracelets VTO, and more.

HandAlignPlugin aligns node’s armature and the hand pose estimated by HandProcessor. Basically, it evaluates positions, rotations, and scales of armature bones based on detected keypoints, then iteratively applies these transforms to bones following skeletal hierarchy. Plugin supports rigs compatible with Clo3D and Marvelous Designer avatars. This is the most common standard of rigs in apparel modeling software. Controlled scene node must contain an armature among its children nodes. Bones of armature must follow Clo3D naming convention and hierarchy. Development of the plugin is not completely finished, especially thumb alignment is not as good as we would like it to be. But it can already be used in cases where precise alignment of thumb is not a hard requirement.

Detection of handedness is now more robust and less noisy, it won’t change between images of a video stream. In all alignment algorithms and plugins we account for handedness to evaluate proper rotation quaternions for left and right hands.

Fingers Detection

Within hand detection we introduce more precise detection of fingers approximating every phalanx by a cylinder. Morphological and statistical analyzes of phalanxes is performed as post-processing stage of hand pose detection. During this stage phalanx edges are detected with sub-pixel accuracy taking into account their occlusions, initial statistical approximation is iteratively refined to minimize error measure. HandProcessor outputs both pixel and metric parametrizations of phalanx cylinders. Output data is filtered using the same approach we apply to reduce temporal noise in detected keypoints, hand detection and phalanx approximation are synchronized. Estimation of phalanxes can be used as approximation of a detected hand in a scene if we render cylinders in place of phalanxes and spheres in place of their joints. Precise detection of phalanxes is crucial for virtual try-on of rings and gloves.

Approximation of phalanxes is enabled by default. In the future, we will make it optional to improve performance in cases where precise approximation isn’t needed. Phalanx detections are output by HandProcessor within HandResult object as phalanxes field. For every phalanx PhalanxDetection defines metric 3D coordinates of its center and two edge points located at the same section as the center.

Hand Model

3D model approximating hand’s fingers by cylinders and spheres can be build using buildGeometry() static method of HandFitPlugin. HandFitPlugin in turn fits HandGeometry into a hand pose and phalanxes estimated by HandProcessor. It evaluates positions, rotations, and scales of phalanxes based on detected keypoints, and then applies these transforms to sub-components of a hand geometry.

HandFitPlugin can be combined with HandAlignPlugin in scenarios where the former approximates finger with higher precision and later is used for coarser approximation of a palm (rest of a hand). This combination is efficient to build and control hand occluder in rings virtual try-on applications where high precision approximation of fingers is required.

RingFitPlugin is a final building block of rings virtual try-on application. This plugin fits a ring object on a pose estimated by HandProcessor. Using tracking data RingFitPlugin estimates the transformation of a ring 3D object fitting it on the selected finger. Plugin supports rings having unit inner diameter and lying in xz plane, with y axis being a center of the ring’s inner circle and x axis pointing in the ring’s head direction. Offset of a ring from the world’s origin along y axis defines how far it will be from the phalanx start. RingFitPlugin is compatible and intended to be used in combination with HandFitPlugin.

Wrist Detection

We completely reworked wrist detection algorithms. New approach is significantly faster and provide more stable and accurate results. GPGPU shaders are significantly re-factored to reduce usage of vector registers and optimize computational flow and memory access pattern.

Wrist detection has been made optional feature of hand tracking to improve performance in cases where wrist tracking is not required. Now it needs to be explicitly enabled by setting wrist parameter to true when initializing a HandProcessor.

New Plugins

ClothTwinPlugin is an equivalent of PoseTwinPlugin but for the newer Clo3D / Marvelous Designer armature. Engeenee SDK supports Clo3D armature as the main rig for virtual try-on applications and improvements of fitting algorithms are developed for this skeletal structure first. For a long time twin functionality was available only for legacy armatures compatible with Mixamo / Ready Player Me. Now visual effects utilising user twins are available with models rigged against more advanced types of armatures. This will simplify design of 3D models and allow to use all recent improvements that are primary developed for the newer armature types. Relative pose of a twin can be adjusted using corresponding methods of the plugin on runtime allowing smooth transitional animations

DelayPlugin is a factory function adding delay to any other plugin. In delayed plugin update() method is called with detection results from one of the previous iterations of a tracking engine. This way we achieve effect of object following its controller with some delay. Delay may be used in combination with various twin plugins to achieve less artificial behaviour of twins that will repeat user movements without absolute synchronization in time and thus look more natural.

VideoMergePlugin is a simple ShaderPlugin that merges input video stream with the background texture. Merge is done by linear interpolation and interpolation weight (alpha) is a tunable input parameter that can be changed on runtime for smooth transition between real and virtual backgrounds.

BgReplacePlugin and VideoMergePlugin are extended with setMirror(boolean) method that allows to mirror virtual background for example when Renderer is in mirror mode. One needs to manually pass proper mirror parameter from a renderer to attached plugins.

Three.js Renderer

This release addresses feature gap between babylon.js and three.js renderers. The list of updates and features ported to three.js renderer:

  • OccluderMaterial similar to the one used in babylon.js renderers.
  • Utility classes and generics for HandResult.
  • Armature utilities to make code base cleaner.
  • HandAlignPlugin.
  • HandFitPlugin
  • Code is refactored to use typed imports reducing size of the bundle.
  • Version of three.js is updated to the latest at the moment.

Refraction Material

RefractionMaterial utilizes real-time ray tracing to calculate refractions of light inside volume of a mesh. It can be used for realistic rendering of diamonds and other gems. Material parameters allow to fine-tune ray tracing mechanics and optimize for quality-performance trade offs. Environment texture is used as a spherical source of light refracting inside the mesh, one can use dynamic texture to reflect surrounding objects of the scene. Other post-processing techniques can be applied on top of initial render to achieve even better quality of gems visualization. Currently RefractionMaterial is available only in three.js renderer; we are planning to implement babylon.js version in the future.

Body and Cloth Fitting

We’ve disabled rescaling of limbs by default. Previously to perfectly fit arms and legs into detected keypoints we rescaled them to align their lengths the way that end of a bone matches the detected joint coordinate. While such approach indeed provides perfect alignment of limb armature, as a side effect arms and legs may scale unnaturally. For example upper arm may become thinner than lower arm that is very noticeable near an elbow. Disabling limbs rescaling makes relative proportions of body parts more natural. Previous limbs rescaling behaviour may be enabled back setting scaleLimbs property of PoseTuneParams.

Additionally, drifting scale error within hand alignment algorithms has been resolved making alignment of hand rig more precise.

Performance Optimizations

  • We’ve added in-place operations to the internal linear algebra library. Their usage in certain algorithms allows to improve performance and use less memory.
  • Trackers use more advanced image pre-processing to increase quality of detection.
  • We’ve compressed binaries of all neural networks, that improves loading and initialization times of virtual try-on applications.
  • Internal dependency on numeric packaged replaced with svd-js.
  • Optimization of imports to reduce bundle size.
  • @geenee/bodyrenderers-three uses the latest version of Three.js.
  • Code of all SDK examples is updated, deprecated plugins are replaced.
  • UI of SDK examples is updated according to the new default style.

API Improvements

  • ShaderProgram supports more types of uniforms, namely integer uniforms are now supported.
  • ResponsiveCanvas is extended with setter and getter for mirror mode that allows to change it on runtime.
  • New ImageCapture video grabber allows to use static images as video sources. It can be used for debug or fine-tune purposes applying virtual try-on or AR effect on a still target.
  • We are replacing Webpack with Vite. Vite.js provides much faster bundling times and much easier to configure. All SDK examples are ported to vite.js.
Tags: