Skip to content

PoseTwinPlugin

Plugin controlling a digital twin

PoseFitPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin’s scene node can be further transformed relative to the initial position - centers of hips are the same. Additionally, callback method can be assigned, it will be called after pose of the twin is set up, hips centers are aligned, and fixed transformations (translation, rotation, scale) are applied. Callback provides more flexible control over twin’s relative pose. Under construction, not well tested after changes.

new PoseTwinPlugin(translation?, rotation?, scale?, callback?, tune?): PoseTwinPlugin

Constructor

Vector3

Relative translation of the twin

Quaternion

Relative rotation of the twin

number

Scale of the twin

(node) => void

Method called after pose is set up

PoseTuneParams

Fine-tuning parameters

PoseTwinPlugin

PoseFitPlugin.constructor

readonly alignScore: 0.9 = 0.90

Pose score threshold

PoseFitPlugin.alignScore


readonly alignVisibility: 0.9 = 0.90

Keypoint visibility threshold

PoseFitPlugin.alignVisibility


protected optional callback: (node) => void

Method called after pose is set up

TransformNode

void


protected cameraAngle: number

Camera vertical angle in radians

PoseFitPlugin.cameraAngle


protected cameraRatio: number

Camera aspect ratio

PoseFitPlugin.cameraRatio


loaded: boolean

Loaded state

PoseFitPlugin.loaded


protected optional node: TransformNode

Attached scene node

PoseFitPlugin.node


protected nodeOrigin: BoneTransform

Origin of node relative to the root bone

PoseFitPlugin.nodeOrigin


ordinal: number

Ordinal number

PoseFitPlugin.ordinal


protected optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

PoseFitPlugin.renderer


protected optional rotation: Quaternion

Relative rotation of the twin


protected optional scale: number

Scale of the twin


protected optional scene: Scene

Reference to a scene instance

PoseFitPlugin.scene


protected optional skeleton: Skeleton

Reference to model’s skeleton

PoseFitPlugin.skeleton


protected optional skeletonNodes: SkeletonNodes

Bones of the model’s rig

PoseFitPlugin.skeletonNodes


protected optional skeletonSizes: SkeletonSizes

Reference sizes of skeleton

PoseFitPlugin.skeletonSizes


protected optional spineCurve: SpineCurve

Shape of spine

PoseFitPlugin.spineCurve


protected optional translation: Vector3

Relative translation of the twin


protected tune: PoseTuneParams

Fine-tuning parameters

PoseFitPlugin.tune

dispose(): void

Dispose video plugin

void

PoseFitPlugin.dispose


protected estimateBones(pose, spineCurve, skeletonSizes): object

Estimate bone positions and orientations

Using detected keypoints approximates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated using kinematic rules and relative positions of adjacent keypoints. Method returns bone position and orientation in global world frame, final relative transformations are found traversing skeleton hierarchy and accumulating transforms of parents.

PosePoints

Pose keypoints

SpineCurve

Reference shape of spine

SkeletonSizes

Reference sizes of skeleton

object

Bone transformations

armL: BoneTransform = armL.arm

armMidL: BoneTransform = armL.armMid

armMidR: BoneTransform = armR.armMid

armR: BoneTransform = armR.arm

footL: BoneTransform = legL.foot

footR: BoneTransform = legR.foot

forearmL: BoneTransform = armL.forearm

forearmMidL: BoneTransform = armL.forearmMid

forearmMidR: BoneTransform = armR.forearmMid

forearmR: BoneTransform = armR.forearm

handL: BoneTransform = armL.hand

handR: BoneTransform = armR.hand

head: BoneTransform

legL: BoneTransform = legL.leg

legR: BoneTransform = legR.leg

neck: BoneTransform

neck1: BoneTransform

pelvis: BoneTransform

root: BoneTransform

shoulderL: BoneTransform = armL.shoulder

shoulderR: BoneTransform = armR.shoulder

spine: BoneTransform

spine1: BoneTransform

spine2: BoneTransform

spine3: BoneTransform

uplegL: BoneTransform = legL.upleg

uplegMidL: BoneTransform = legL.uplegMid

uplegMidR: BoneTransform = legR.uplegMid

uplegR: BoneTransform = legR.upleg

PoseFitPlugin.estimateBones


load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.

Renderer<PoseResult>

Renderer this plugin is attached to

Promise<void>

Promise resolving when initialization is finished

PoseFitPlugin.load


setCallback(callback): void

undefined | (node) => void

void


setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

TransformNode

Scene node to attach

void

PoseFitPlugin.setNode


setRotation(rotation?): void

Quaternion

void


setScale(scale?): void

number

void


setTranslation(translation?): void

Vector3

void


setupCamera(ratio, angle): void

Set camera parameters

number

Aspect ration of input video

number

Vertical field of view in radians

void

PoseFitPlugin.setupCamera


setupVideo(size): void

Set video size

Could be overridden to adjust plugin’s pipeline.

Size

Resolution of input video

void

PoseFitPlugin.setupVideo


unload(): void

Reset plugin

Clears internal state and frees all resources allocated in load().

void

PoseFitPlugin.unload


update(result, stream): Promise<void>

Update skeleton of the scene node

Evaluates positions, rotations and scales of node bones based on estimation of 3D keypoints, then applies these transformations to bones following hierarchy of armature. To improve accuracy of alignment detected 3D points and skeleton sizes are mutually adjusted to fit each other. One can override this method to further tune model’s rig using provided estimations of bones as a starting point. Simply call await super.update(result, stream); and use PoseFitPlugin#skeletonNodes member storing refs to all bones of the skeleton to access transformations.

PoseResult

Pose estimation results

HTMLCanvasElement

Captured video frame

Promise<void>

Promise resolving when update is finished

PoseFitPlugin.update


protected updateArm(bones, points): void

Update arm rig (see update)

Arm bones and transforms

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

Arm keypoints

PosePoint

PosePoint

void

PoseFitPlugin.updateArm


protected updateLeg(bones, points): void

Update leg rig (see update)

Leg bones and transforms

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

{ bone: TransformNode; transform: BoneTransform; }

TransformNode

BoneTransform

Leg keypoints

PosePoint

PosePoint

void

PoseFitPlugin.updateLeg


protected updateSpine(transforms, bones): void

Update spine rig (see update)

Estimated transformations

BoneTransform = armL.arm

BoneTransform = armL.armMid

BoneTransform = armR.armMid

BoneTransform = armR.arm

BoneTransform = legL.foot

BoneTransform = legR.foot

BoneTransform = armL.forearm

BoneTransform = armL.forearmMid

BoneTransform = armR.forearmMid

BoneTransform = armR.forearm

BoneTransform = armL.hand

BoneTransform = armR.hand

BoneTransform = ...

BoneTransform = legL.leg

BoneTransform = legR.leg

BoneTransform = ...

BoneTransform = ...

BoneTransform = ...

BoneTransform = ...

BoneTransform = armL.shoulder

BoneTransform = armR.shoulder

BoneTransform = ...

BoneTransform = ...

BoneTransform = ...

BoneTransform = ...

BoneTransform = legL.upleg

BoneTransform = legL.uplegMid

BoneTransform = legR.uplegMid

BoneTransform = legR.upleg

SkeletonNodes

Spine bones

void

PoseFitPlugin.updateSpine