Skip to main content

Class: PoseTwinPlugin

@geenee/bodyrenderers-three.PoseTwinPlugin

Plugin rendering a digital twin

PoseAlignPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin's scene node can be further transformed relative to the initial position - centers of hips are the same.

Hierarchy

Constructors

constructor

new PoseTwinPlugin(node?, translation?, rotation?, scale?, tune?)

Constructor

Parameters

NameTypeDescription
node?Object3D<Object3DEventMap>Scene node to attach
translation?Vector3Relative translation of the twin
rotation?QuaternionRelative rotation of the twin
scale?numberScale of the twin
tunePoseTuneParamsFine-tuning parameters

Overrides

PoseAlignPlugin.constructor

Properties

alignScore

Readonly alignScore: 0.9

Pose score threshold

Inherited from

PoseAlignPlugin.alignScore


alignVisibility

Readonly alignVisibility: 0.9

Keypoint visibility threshold

Inherited from

PoseAlignPlugin.alignVisibility


avatarLength

Protected avatarLength: number = 1

Reference length of the model

Inherited from

PoseAlignPlugin.avatarLength


loaded

loaded: boolean

Loaded state

Inherited from

PoseAlignPlugin.loaded


node

Protected Optional node: Object3D<Object3DEventMap>

Scene node to attach

Inherited from

PoseAlignPlugin.node


ordinal

ordinal: number

Ordinal number

Inherited from

PoseAlignPlugin.ordinal


renderer

Protected Optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

Inherited from

PoseAlignPlugin.renderer


rotation

Protected Optional rotation: Quaternion

Relative rotation of the twin


scale

Protected Optional scale: number

Scale of the twin


scene

Protected Optional scene: Scene

Reference to a scene instance

Inherited from

PoseAlignPlugin.scene


skeleton

Protected Optional skeleton: Skeleton

Reference to model's skeleton

Inherited from

PoseAlignPlugin.skeleton


skeletonNodes

Protected Optional skeletonNodes: SkeletonNodes

Bones of the model's rig

Inherited from

PoseAlignPlugin.skeletonNodes


spineCurve

Protected Optional spineCurve: SpineCurve

Shape of spine

Inherited from

PoseAlignPlugin.spineCurve


translation

Protected Optional translation: Vector3

Relative translation of the twin


tune

Protected tune: PoseTuneParams

Fine-tuning parameters

Inherited from

PoseAlignPlugin.tune

Methods

alignBone

Protected alignBone(transform, bone): void

Set bone aligning transformation

Overridden to assign only relative rotation.

Parameters

NameTypeDescription
transformBoneTransformGlobal bone position and rotation
boneBone<Object3DEventMap>Reference to bone instance

Returns

void

Overrides

PoseAlignPlugin.alignBone


dispose

dispose(): void

Dispose video plugin

Returns

void

Inherited from

PoseAlignPlugin.dispose


estimateBones

Protected estimateBones(points, spineCurve): SkeletonTransforms

Estimate bone positions and axes

Based on detected keypoints estimates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone's rotation is defined by its axes that are evaluated from relative positions of adjacent keypoints. Method returns only bone position and orientation axis, final transformation of any bone can be found using the next bone in hierarchy.

Parameters

NameTypeDescription
pointsPosePointsPose keypoints
spineCurveSpineCurveShape of spine

Returns

SkeletonTransforms

Bone transformations

Inherited from

PoseAlignPlugin.estimateBones


load

load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin's constructor). Precalculates geometrical parameters of skeleton.

Parameters

NameTypeDescription
rendererRenderer<PoseResult>Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Inherited from

PoseAlignPlugin.load


setNode

setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

Parameters

NameTypeDescription
node?Object3D<Object3DEventMap>Scene node to attach

Returns

void

Inherited from

PoseAlignPlugin.setNode


setupCamera

setupCamera(ratio, angle): void

Set camera parameters

Could be overridden to adjust plugin's pipeline.

Parameters

NameTypeDescription
rationumberAspect ration of input video
anglenumberVertical field of view in radians

Returns

void

Inherited from

PoseAlignPlugin.setupCamera


setupVideo

setupVideo(size): void

Set video size

Could be overridden to adjust plugin's pipeline.

Parameters

NameTypeDescription
sizeSizeResolution of input video

Returns

void

Inherited from

PoseAlignPlugin.setupVideo


unload

unload(): void

Reset plugin

Removes the attached node.

Returns

void

Inherited from

PoseAlignPlugin.unload


update

update(result, stream): Promise<void>

Update skeleton of the scene node

Method is extended to set twin's translation, rotation, and scale relative to estimated pose.

Parameters

NameTypeDescription
resultPoseResultPose estimation results
streamHTMLCanvasElementCaptured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

PoseAlignPlugin.update


updateHandL

Protected updateHandL(anchors, points): void

Update left hand skeleton

Parameters

NameTypeDescription
anchorsSkeletonTransformsPositions and axes of bones
pointsPosePointsPose keypoints

Returns

void

Inherited from

PoseAlignPlugin.updateHandL


updateHandR

Protected updateHandR(anchors, points): void

Update right hand skeleton

Parameters

NameTypeDescription
anchorsSkeletonTransformsPositions and axes of bones
pointsPosePointsPose keypoints

Returns

void

Inherited from

PoseAlignPlugin.updateHandR


updateLegL

Protected updateLegL(anchors, points): void

Update left leg skeleton

Parameters

NameTypeDescription
anchorsSkeletonTransformsPositions and axes of bones
pointsPosePointsPose keypoints

Returns

void

Inherited from

PoseAlignPlugin.updateLegL


updateLegR

Protected updateLegR(anchors, points): void

Update right leg skeleton

Parameters

NameTypeDescription
anchorsSkeletonTransformsPositions and axes of bones
pointsPosePointsPose keypoints

Returns

void

Inherited from

PoseAlignPlugin.updateLegR


updateSpine

Protected updateSpine(anchors): void

Update spine skeleton

Overridden to ignore relative scaling.

Parameters

NameTypeDescription
anchorsSkeletonTransformsPositions and axes of bones

Returns

void

Overrides

PoseAlignPlugin.updateSpine