Three.js Renderer
Three.js Renderer
Section titled “Three.js Renderer”@geenee/bodyrenderers-three module utilizes three.js rendering engine for visualization. Babylon.js renderers and plugins can be found in @geenee/bodyrenderers-babylon package.
Basics
Section titled “Basics”Set of abstract classes that specialize generic renderers and plugins for PoseProcessor and FaceProcessor. These classes are used as base parents to simplify API, they do not implement any logic or visualization.
- PoseRenderer - Abstract renderer for PoseProcessor
- PosePlugin - Abstract plugin for PoseRenderer
- FaceRenderer - Abstract renderer for FaceProcessor
- FacePlugin - Abstract plugin for FaceRenderer
- ThreeRenderer - Generic three.js renderer
- ThreePlugin - Generic plugin for ThreeRenderer
Pose Tracking
Section titled “Pose Tracking”Universal plugin aligning node’s rig and pose estimated by @geenee/bodyprocessors!PoseProcessor. It’s a base of try-on, twin, and other plugins. You can use this class as a starting point and customize alignment method or add features. Basically, PoseBFitPlugin evaluates positions and rotations of armature bones based on 3D pose keypoints, then applies these transforms to bones following the armature hierarchy. Plugin supports model rigs compatible with Mixamo armature, e.g. any model from Mixamo library or Ready Player Me avatars. This is common standard of armature/skeleton for human-like / anthropomorphic models supported by many game/render engines. The scene node must contain an armature among its children. Armature’s bones must follow Mixamo / RPM naming convention. Models rigged and skinned manually or using Mixamo tool can variate depending on anthropomorphous topology of the model. For example animated characters can have disproportional body parts like a much bigger head or longer arms. In such cases PoseBFitPlugin can apply number of fine-tuning adjustments to basic alignment improving model fitting or making it look more natural. PoseTuneParams explains tuning options. As an example turning off adjustment of spine curvature gives better results in virtual garment try-on experiences, while for full-body avatar overlaying it can provide more natural look. Depending on the use case and model’s topology you can try to tune different options and see what works better in practice. By default the plugin is fine-tuned for RPM avatars so you can simply replace person with the avatar model in the scene.
PoseBFitPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin’s scene node can be further transformed relative to the initial position - centers of hips are the same.
Pose Tracking Example
Section titled “Pose Tracking Example”- Download pose tracking example for three.js
- Get access tokens on your account page.
- Replace placeholder in
.npmrc
file with your personal NPM token. - Run
npm install
to install all dependency packages. - In
src/index.ts
set your SDK access tokens (replace stubs). - Run
npm run start
ornpm run start:https
. - Open
http(s)://localhost:3000
url in a browser. - That’s it, you first pose tracking AR application is ready.
Preparing models
Section titled “Preparing models”Guides on preparing models for pose tracking:
Face Tracking
Section titled “Face Tracking”Plugin attaches provided scene node to the head. Pose of the node (translation + rotation + scale) continuously updates according to pose estimation by @geenee/bodyprocessors!FaceProcessor. Children nodes inherently include this transform. The node can be seen as a virtual placeholder for real object. It’s recommended to attach top-level nodes that don’t include transforms relative to parent, otherwise head transform that is a pose in the world frame will be applied on top of them (will be treated as relative instead of absolute). Optionally anisotropic fine-tuning of the scale can be applied. In this case model will additionally adapt to shape of the face. If face isn’t detected by FaceProcessor plugin recursively hides the node.
Download reference face model: face.glb.
Simplified implementation:
async update(result: FaceResult, stream: HTMLCanvasElement) { if (!this.loaded) return; const { transform } = result; if (!transform) { this.node.visible = false; return super.update(result, stream); } // Mesh transformation const translation = new three.Vector3(...transform.translation) const uniformScale = new three.Vector3().setScalar(transform.scale); const shapeScale = new three.Vector3( ...transform.shapeScale).multiplyScalar(transform.scale) const rotation = new three.Quaternion(...transform.rotation); // Align node with the face this.node.visible = true; this.node.setRotationFromQuaternion(rotation); this.node.position.copy(translation); this.node.scale.copy(this.shapeScale ? shapeScale : uniformScale); // Render return super.update(result, stream);}
Plugin attaches provided node to the face point. Pose of the node (translation + rotation + scale) continuously updates according to pose estimation by @geenee/bodyprocessors!FaceProcessor. Children nodes inherently include this transform. The node can be seen as a virtual placeholder for real object. It’s recommended to attach top-level nodes that don’t include transforms relative to parent, otherwise head transform that is a pose in the world frame will be applied on top of them (will be treated as relative instead of absolute). Optionally anisotropic fine-tuning of the scale can be applied. In this case model will additionally adapt to shape of the face. If face isn’t detected by FaceProcessor plugin recursively hides the node.
Download reference face model: face.glb.
Simplified implementation:
async update(result: FaceResult, stream: HTMLCanvasElement) { if (!this.loaded) return; const { transform } = result; if (!transform) { this.node.visible = false; return super.update(result, stream); } // Mesh transformation const translation = new three.Vector3(...transform.translation) const uniformScale = new three.Vector3().setScalar(transform.scale); const shapeScale = new three.Vector3( ...transform.shapeScale).multiplyScalar(transform.scale) const rotation = new three.Quaternion(...transform.rotation); // Align node with the face this.node.visible = true; this.node.setRotationFromQuaternion(rotation); this.node.position.copy(translation); this.node.scale.copy(this.shapeScale ? shapeScale : uniformScale); // Render return super.update(result, stream);}
Adds a Mesh object to the scene that reflects detected face mesh. FaceMaskPlugin creates Mesh and defines indices, uvs and normals of vertices in load(), while vertex positions are updated in update() according to current face tracking estimations. The plugin uses MeshStandardMaterial with a diffuse texture provided as image url.
Download face UV map: faceuv.png
Simplified implementation:
async load(scene?: three.Scene) { if (this.loaded || !scene) return; // Mask geometry const geometry = new three.BufferGeometry(); geometry.setIndex(meshTriangles); geometry.setAttribute("position", new three.Float32BufferAttribute(new Float32Array(468 * 3), 3)); geometry.setAttribute("uv", new three.Float32BufferAttribute(meshUV.flat(), 2)); geometry.computeVertexNormals(); // Texture const texture = new three.TextureLoader().load(this.url); texture.flipY = false; const material = new three.MeshStandardMaterial({ map: texture, transparent: true }); // Add mask to the scene this.mask = new three.Mesh(geometry, material); scene.add(this.mask); return super.load(scene);}
async update(result: FaceResult, stream: HTMLCanvasElement) { if (!this.loaded) return; if (!this.mask) return super.update(result, stream); const { metric } = result; if (!metric) { this.mask.visible = false; return super.update(result, stream); } // Update mesh coordinates this.mask.visible = true; let position = this.mask.geometry.getAttribute("position"); metric.forEach((p, i) => position.setXYZ(i, p[0], p[1], p[2])); this.mask.geometry.computeVertexNormals(); position.needsUpdate = true; // Render return super.update(result, stream);}
Face Tracking Example
Section titled “Face Tracking Example”- Download face tracking example for three.js
- Get access tokens on your account page.
- Replace placeholder in
.npmrc
file with your personal NPM token. - Run
npm install
to install all dependency packages. - In
src/index.ts
set your SDK access tokens (replace stubs). - Run
npm run start
ornpm run start:https
. - Open
http(s)://localhost:3000
url in a browser. - That’s it, you first face tracking AR application is ready.
Preparing Models
Section titled “Preparing Models”Guides on preparing models for face tracking:
Hand Tracking
Section titled “Hand Tracking”Universal plugin aligning node’s armature and the hand pose estimated by @geenee/bodyprocessors!HandProcessor. Basically, HandFitPlugin evaluates positions and rotations of armature bones based on detected keypoints, then applies these transforms to bones following the armature hierarchy. Plugin supports rigs compatible with Clo3D and Marvelous Designer avatars. This is the most common standard of rigs in cloth/apparel modeling software. Controlled scene node must contain an armature among its children nodes. Bones of armature must follow Clo3D naming convention and hierarchy.
Universal plugin fitting FingersGeometry into a pose estimated by @geenee/bodyprocessors!HandProcessor. FingersFitPlugin evaluates positions, rotations and scales of phalanxes based on detected keypoints, and then applies these transforms to sub-components of a hand geometry. Use FingersFitPlugin#buildGeometry to build geometry.
Plugin fitting a ring object on a pose estimated by @geenee/bodyprocessors!HandProcessor. Using tracking data RingFitPlugin estimates the transformation of a ring 3d object fitting it on the selected finger. Plugin supports rings having unit inner diameter and lying in xz plane, with y axis being a center of the ring’s inner circle and x axis pointing in the ring’s head direction. Offset of a ring from world’s origin along y axis defines how far it will be from the phalanx start. See FingersFitPlugin for additional details.
Hand Tracking Example
Section titled “Hand Tracking Example”- Download hand tracking example for three.js
- Get access tokens on your account page.
- Replace placeholder in
.npmrc
file with your personal NPM token. - Run
npm install
to install all dependency packages. - In
src/index.ts
set your SDK access tokens (replace stubs). - Run
npm run start
ornpm run start:https
. - Open
http(s)://localhost:3000
url in a browser. - That’s it, you first hand tracking AR application is ready.
Occluders
Section titled “Occluders”Applying OccluderMaterial to a mesh makes it an occluder. It’s recommended to render occluder meshes prior to visible meshes of a scene setting their renderOrder property to -1.
Example of usage:
setOccluders(scene: babylon.Scene) { const body = scene.getMeshByName("Body"); const head = scene.getMeshByName("Head"); if (!body && !head) return; const material = new OccluderMaterial("Occluder", scene); if (body) body.material = material; if (head) head.material = material;}
Extensions
Section titled “Extensions”Refraction material utilizes ray tracing approach to implement refractions of light inside a mesh volume. It is used for rendering of gems, diamons or glass. Several parameters fine-tune ray tracing mechanics and optimize for quality-performance trade offs. Environment texture is used as a sperical source of light refracting inside the mesh, one can use dynamic texture to reflect surrounding objects of the scene.