Skip to content
Engeenee
Blog

@geenee/bodyrenderers-three

Renderer

@geenee/armature!Renderer is the core visualization and logical part of any application. It’s attached to the @geenee/armature!Engine. Basically, renders define two methods load() and update(). The first one is used to initialize assets and prepare the scene (lightning, environment map). The second is used to update the scene according to results of video processing. This’s where all the logic happens. Renderers can be extended with plugins. Plugins do simple rendering task, for example add object to that follows the head or render avatar overlay.

This package provides set of ready-made renderers and plugins to simplify development of applications. They can be used as both atomic building blocks or you can use them as starting points, inherit and override / extend class functionality. By extending @geenee/armature!Renderer#load and @geenee/armature!Renderer#update of a Renderer or Plugin’s @geenee/armature!ScenePlugin#load and @geenee/armature!ScenePlugin#update you can add any custom logic, interactions, animations, post-processing, effects, gesture recognition, etc.

Module utilizes three.js rendering engine for visualization. Babylon.js renderers and plugins can be found in @geenee/bodyrenderers-babylon! package.

Basics

Set of abstract classes that specialize generic renderers and plugins for @geenee/bodyprocessors!PoseProcessor and @geenee/bodyprocessors!FaceProcessor. These classes are used as base parents to simplify API, they do not implement any logic or visualization.

Pose Tracking

PoseAlignPlugin

Universal plugin aligning node’s rig and pose estimated by @geenee/bodyprocessors!PoseProcessor. It’s a base of try-on, twin, and other plugins. You can use this class as a starting point and customize alignment method or add features. Basically, PoseAlignPlugin evaluates positions and rotations of armature bones based on 3D pose keypoints, then applies these transforms to bones following the armature hierarchy. Plugin supports model rigs compatible with Mixamo armature, e.g. any model from Mixamo library or Ready Player Me avatars. This is common standard of armature/skeleton for human-like / anthropomorphic models supported by many game/render engines. The scene node must contain an armature among its children. Armature’s bones must follow Mixamo / RPM naming convention. Models rigged and skinned manually or using Mixamo tool can variate depending on anthropomorphous topology of the model. For example animated characters can have disproportional body parts like a much bigger head or longer arms. In such cases PoseAlignPlugin can apply number of fine-tuning adjustments to basic alignment improving model fitting or making it look more natural. PoseTuneParams explains tuning options. As an example turning off adjustment of spine curvature gives better results in virtual garment try-on experiences, while for full-body avatar overlaying it can provide more natural look. Depending on the use case and model’s topology you can try to tune different options and see what works better in practice. By default the plugin is fine-tuned for RPM avatars so you can simply replace person with the avatar model in the scene.

PoseOutfitPlugin (Deprecated)

PoseOutfitPlugin is extension of PoseAlignPlugin that allows to specify body meshes of the avatar’s node as occluders and optionally hide some child meshes (parts). It’s a good starting point for virtual try-on applications. OutfitParams defines available options of outfit. You can download any Ready Player Me avatar which outfit similar to final result, edit its outfit, re-skin model if necessary. Then simply use this plugin to build try-on app. Armature bones must follow Mixamo / RPM naming convention.

Deprecated: Use OccluderMaterial or OccluderMaskPlugin directly to make meshes of the node into occluders and hide them manually by setEnabled(), this’s more flexible approach.

PoseTwinPlugin

PoseAlignPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin’s scene node can be further transformed relative to the initial position - centers of hips are the same.

Pose Tracking Example

  • Download pose tracking example for three.js
  • Get access tokens on your account page.
  • Replace placeholder in .npmrc file with your personal NPM token.
  • Run npm install to install all dependency packages.
  • In src/index.ts set your SDK access tokens (replace stubs).
  • Run npm run start or npm run start:https.
  • Open http(s)://localhost:3000 url in a browser.
  • That’s it, you first pose tracking AR application is ready.

Preparing models

Guides on preparing models for pose tracking:

Face Tracking

HeadTrackPlugin

Plugin attaches provided scene node to the head. Pose of the node (translation + rotation + scale) continuously updates according to pose estimation by @geenee/bodyprocessors!FaceProcessor. Children nodes inherently include this transform. The node can be seen as a virtual placeholder for real object. It’s recommended to attach top-level nodes that don’t include transforms relative to parent, otherwise head transform that is a pose in the world frame will be applied on top of them (will be treated as relative instead of absolute). Optionally anisotropic fine-tuning of the scale can be applied. In this case model will additionally adapt to shape of the face. If face isn’t detected by FaceProcessor plugin recursively hides the node.

Download reference face model: face.glb.

Simplified implementation:

async update(result: FaceResult, stream: HTMLCanvasElement) {
    if (!this.loaded)
        return;
    const { transform } = result;
    if (!transform) {
        this.node.visible = false;
        return super.update(result, stream);
    }
    // Mesh transformation
    const translation = new three.Vector3(...transform.translation)
    const uniformScale = new three.Vector3().setScalar(transform.scale);
    const shapeScale = new three.Vector3(
        ...transform.shapeScale).multiplyScalar(transform.scale)
    const rotation = new three.Quaternion(...transform.rotation);
    // Align node with the face
    this.node.visible = true;
    this.node.setRotationFromQuaternion(rotation);
    this.node.position.copy(translation);
    this.node.scale.copy(this.shapeScale ? shapeScale : uniformScale);
    // Render
    return super.update(result, stream);
}

FaceTrackPlugin

Plugin attaches provided node to the face point. Pose of the node (translation + rotation + scale) continuously updates according to pose estimation by @geenee/bodyprocessors!FaceProcessor. Children nodes inherently include this transform. The node can be seen as a virtual placeholder for real object. It’s recommended to attach top-level nodes that don’t include transforms relative to parent, otherwise head transform that is a pose in the world frame will be applied on top of them (will be treated as relative instead of absolute). Optionally anisotropic fine-tuning of the scale can be applied. In this case model will additionally adapt to shape of the face. If face isn’t detected by FaceProcessor plugin recursively hides the node.

Download reference face model: face.glb.

Simplified implementation:

async update(result: FaceResult, stream: HTMLCanvasElement) {
    if (!this.loaded)
        return;
    const { transform } = result;
    if (!transform) {
        this.node.visible = false;
        return super.update(result, stream);
    }
    // Mesh transformation
    const translation = new three.Vector3(...transform.translation)
    const uniformScale = new three.Vector3().setScalar(transform.scale);
    const shapeScale = new three.Vector3(
        ...transform.shapeScale).multiplyScalar(transform.scale)
    const rotation = new three.Quaternion(...transform.rotation);
    // Align node with the face
    this.node.visible = true;
    this.node.setRotationFromQuaternion(rotation);
    this.node.position.copy(translation);
    this.node.scale.copy(this.shapeScale ? shapeScale : uniformScale);
    // Render
    return super.update(result, stream);
}

FaceMaskPlugin

Adds a Mesh object to the scene that reflects detected face mesh. FaceMaskPlugin creates Mesh and defines indices, uvs and normals of vertices in load(), while vertex positions are updated in update() according to current face tracking estimations. The plugin uses MeshStandardMaterial with a diffuse texture provided as image url.

Download face UV map: faceuv.png

Simplified implementation:

async load(scene?: three.Scene) {
    if (this.loaded || !scene)
        return;
    // Mask geometry
    const geometry = new three.BufferGeometry();
    geometry.setIndex(meshTriangles);
    geometry.setAttribute("position",
        new three.Float32BufferAttribute(new Float32Array(468 * 3), 3));
    geometry.setAttribute("uv",
        new three.Float32BufferAttribute(meshUV.flat(), 2));
    geometry.computeVertexNormals();
    // Texture
    const texture = new three.TextureLoader().load(this.url);
    texture.flipY = false;
    const material = new three.MeshStandardMaterial({
        map: texture, transparent: true
    });
    // Add mask to the scene
    this.mask = new three.Mesh(geometry, material);
    scene.add(this.mask);
    return super.load(scene);
}


async update(result: FaceResult, stream: HTMLCanvasElement) {
    if (!this.loaded)
        return;
    if (!this.mask)
        return super.update(result, stream);
    const { metric } = result;
    if (!metric) {
        this.mask.visible = false;
        return super.update(result, stream);
    }
    // Update mesh coordinates
    this.mask.visible = true;
    let position = this.mask.geometry.getAttribute("position");
    metric.forEach((p, i) =>
        position.setXYZ(i, p[0], p[1], p[2]));
    this.mask.geometry.computeVertexNormals();
    position.needsUpdate = true;
    // Render
    return super.update(result, stream);
}

Face Tracking Example

  • Download face tracking example for three.js
  • Get access tokens on your account page.
  • Replace placeholder in .npmrc file with your personal NPM token.
  • Run npm install to install all dependency packages.
  • In src/index.ts set your SDK access tokens (replace stubs).
  • Run npm run start or npm run start:https.
  • Open http(s)://localhost:3000 url in a browser.
  • That’s it, you first face tracking AR application is ready.

Preparing Models

Guides on preparing models for face tracking:

Occluders

OccluderPlugin

Plugin making provided node an occluder. Usually node is a base mesh (average approximation) of a body representing its real counterpart in a scene. Occluders are not rendered by themselves but still participate in occlusion queries. This is achieved by setting colorWrite=false to all materials of node’s meshes. This flag tells rendering engine to not write to color buffer but still write to depth buffer. Then meshes are effectively not rendered (fragment color write is skipped) and only occlude all other meshes of the scene (during depth test).

Simplified implementation:

async load(scene?: three.Scene) {
    if (this.loaded || !scene)
        return;
    // Occluder material
    this.node.traverse((child) => {
        if (child instanceof three.Mesh)
            child.material.colorWrite = false;
        child.renderOrder = -1;
    });
    return super.load(scene);
}

Namespaces

Classes

Interfaces

Type Aliases

Variables

Functions