Class: PoseTwinPlugin

@geenee/bodyrenderers-babylon.PoseTwinPlugin

Plugin rendering a digital twin

PoseAlignPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin's scene node can be further transformed relative to the initial position - centers of hips are the same.

Hierarchy

• PoseAlignPlugin

  ↳ PoseTwinPlugin

Constructors

constructor

• new PoseTwinPlugin(node?, translation?, rotation?, scale?, tune?)

Constructor

Parameters

Name	Type	Description
node?	TransformNode	Scene node to attach
translation?	Vector3	Relative translation of the twin
rotation?	Quaternion	Relative rotation of the twin
scale?	number	Scale of the twin
tune	PoseTuneParams	Fine-tuning parameters

Overrides

PoseAlignPlugin.constructor

Properties

alignScore

• Readonly alignScore: 0.9

Pose score threshold

Inherited from

PoseAlignPlugin.alignScore

---

alignVisibility

• Readonly alignVisibility: 0.9

Keypoint visibility threshold

Inherited from

PoseAlignPlugin.alignVisibility

---

cameraAngle

• Protected cameraAngle: number

Camera vertical angle in radians

Inherited from

PoseAlignPlugin.cameraAngle

---

cameraRatio

• Protected cameraRatio: number

Camera aspect ratio

Inherited from

PoseAlignPlugin.cameraRatio

---

loaded

• loaded: boolean

Loaded state

Inherited from

PoseAlignPlugin.loaded

---

node

• Protected Optional node: TransformNode

Scene node to attach

Inherited from

PoseAlignPlugin.node

---

nodeOrigin

• Protected nodeOrigin: BoneTransform

Origin of node relative to the root bone

Inherited from

PoseAlignPlugin.nodeOrigin

---

ordinal

• ordinal: number

Ordinal number

Inherited from

PoseAlignPlugin.ordinal

---

renderer

• Protected Optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

Inherited from

PoseAlignPlugin.renderer

---

rotation

• Protected Optional rotation: Quaternion

Relative rotation of the twin

---

scale

• Protected Optional scale: number

Scale of the twin

---

scene

• Protected Optional scene: Scene

Reference to a scene instance

Inherited from

PoseAlignPlugin.scene

---

skeleton

• Protected Optional skeleton: Skeleton

Reference to model's skeleton

Inherited from

PoseAlignPlugin.skeleton

---

skeletonNodes

• Protected Optional skeletonNodes: SkeletonNodes

Bones of the model's rig

Inherited from

PoseAlignPlugin.skeletonNodes

---

skeletonSizes

• Protected Optional skeletonSizes: SkeletonSizes

Reference sizes of skeleton

Inherited from

PoseAlignPlugin.skeletonSizes

---

spineCurve

• Protected Optional spineCurve: SpineCurve

Shape of spine

Inherited from

PoseAlignPlugin.spineCurve

---

translation

• Protected Optional translation: Vector3

Relative translation of the twin

---

tune

• Protected tune: PoseTuneParams

Fine-tuning parameters

Inherited from

PoseAlignPlugin.tune

Methods

alignBone

▸ Protected alignBone(transform, bone): void

Set bone aligning transformation

Overridden to assign only relative rotation.

Parameters

Name	Type	Description
transform	BoneTransform	Global bone position and rotation
bone	TransformNode	Reference to bone instance

Returns

void

Overrides

PoseAlignPlugin.alignBone

---

dispose

▸ dispose(): void

Dispose video plugin

Returns

void

Inherited from

PoseAlignPlugin.dispose

---

estimateBones

▸ Protected estimateBones(pose, spineCurve, skeletonSizes): SkeletonTransforms

Estimate bone positions and axes

Based on detected keypoints estimates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone's rotation is defined by its axes that are evaluated from relative positions of adjacent keypoints. Method returns only bone position and orientation axis, final transformation of any bone can be found using the next bone in hierarchy.

Parameters

Name	Type	Description
pose	PosePoints	Pose keypoints
spineCurve	SpineCurve	Shape of spine
skeletonSizes	SkeletonSizes	-

Returns

SkeletonTransforms

Bone transformations

Inherited from

PoseAlignPlugin.estimateBones

---

load

▸ load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin's constructor). Precalculates geometrical parameters of skeleton.

Parameters

Name	Type	Description
renderer	Renderer<PoseResult>	Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Inherited from

PoseAlignPlugin.load

---

setNode

▸ setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

Parameters

Name	Type	Description
node?	TransformNode	Scene node to attach

Returns

void

Inherited from

PoseAlignPlugin.setNode

---

setupCamera

▸ setupCamera(ratio, angle): void

Set camera parameters

Parameters

Name	Type	Description
ratio	number	Aspect ration of input video
angle	number	Vertical field of view in radians

Returns

void

Inherited from

PoseAlignPlugin.setupCamera

---

setupVideo

▸ setupVideo(size): void

Set video size

Could be overridden to adjust plugin's pipeline.

Parameters

Name	Type	Description
size	Size	Resolution of input video

Returns

void

Inherited from

PoseAlignPlugin.setupVideo

---

unload

▸ unload(): void

Reset plugin

Removes the attached node.

Returns

void

Inherited from

PoseAlignPlugin.unload

---

update

▸ update(result, stream): Promise<void>

Update skeleton of the scene node

Method is extended to set twin's translation, rotation, and scale relative to estimated pose.

Parameters

Name	Type	Description
result	PoseResult	Pose estimation results
stream	HTMLCanvasElement	Captured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

PoseAlignPlugin.update

---

updateArm

▸ Protected updateArm(bones, points): void

Update arm rig (see update)

Parameters

Name	Type	Description
bones	Object	Arm bones and transforms
bones.arm	Object	-
bones.arm.bone	TransformNode	-
bones.arm.transform	BoneTransform	-
bones.forearm	Object	-
bones.forearm.bone	TransformNode	-
bones.forearm.transform	BoneTransform	-
bones.hand	Object	-
bones.hand.bone	TransformNode	-
bones.hand.transform	BoneTransform	-
points	Object	Arm ke ypoints
points.elbow	PosePoint	-
points.shoulder	PosePoint	-

Returns

void

Inherited from

PoseAlignPlugin.updateArm

---

updateLeg

▸ Protected updateLeg(bones, points): void

Update leg rig (see update)

Parameters

Name	Type	Description
bones	Object	Leg bones and transforms
bones.foot	Object	-
bones.foot.bone	TransformNode	-
bones.foot.transform	BoneTransform	-
bones.leg	Object	-
bones.leg.bone	TransformNode	-
bones.leg.transform	BoneTransform	-
bones.toe	Object	-
bones.toe.bone	TransformNode	-
bones.toe.transform	BoneTransform	-
bones.upleg	Object	-
bones.upleg.bone	TransformNode	-
bones.upleg.transform	BoneTransform	-
points	Object	Leg key points
points.ankle	PosePoint	-
points.hip	PosePoint	-
points.knee	PosePoint	-

Returns

void

Inherited from

PoseAlignPlugin.updateLeg

---

updateSpine

▸ Protected updateSpine(anchors): void

Update spine skeleton

Overridden to ignore relative scaling.

Parameters

Name	Type	Description
anchors	SkeletonTransforms	Positions and axes of bones

Returns

void

Overrides

PoseAlignPlugin.updateSpine