PoseTwinPlugin

Plugin controlling a digital twin

PoseAlignPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin’s scene node can be further transformed relative to the initial position - centers of hips are the same. Under construction, not well tested after changes.

Experimental

This API should not be used in production and may be trimmed from a public release.

Extends

• PoseAlignPlugin

Constructors

new PoseTwinPlugin()

new PoseTwinPlugin(node?, translation?, rotation?, scale?, tune?): PoseTwinPlugin

Constructor

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• node?: TransformNode

Scene node to attach

• translation?: Vector3

Relative translation of the twin

• rotation?: Quaternion

Relative rotation of the twin

• scale?: number

Scale of the twin

• tune?: PoseTuneParams = ...

Fine-tuning parameters

Returns

PoseTwinPlugin

Overrides

PoseAlignPlugin.constructor

Properties

alignScore

readonly alignScore: 0.9 = 0.90

Pose score threshold

Inherited from

PoseAlignPlugin.alignScore

---

alignVisibility

readonly alignVisibility: 0.9 = 0.90

Keypoint visibility threshold

Inherited from

PoseAlignPlugin.alignVisibility

---

cameraAngle

protected cameraAngle: number

Camera vertical angle in radians

Inherited from

PoseAlignPlugin.cameraAngle

---

cameraRatio

protected cameraRatio: number

Camera aspect ratio

Inherited from

PoseAlignPlugin.cameraRatio

---

loaded

loaded: boolean

Loaded state

Inherited from

PoseAlignPlugin.loaded

---

node?

protected optional node: TransformNode

Scene node to attach

Inherited from

PoseAlignPlugin.node

---

nodeOrigin

protected nodeOrigin: BoneTransform

Origin of node relative to the root bone

Inherited from

PoseAlignPlugin.nodeOrigin

---

ordinal

ordinal: number

Ordinal number

Inherited from

PoseAlignPlugin.ordinal

---

renderer?

protected optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

Inherited from

PoseAlignPlugin.renderer

---

rotation?

protected optional rotation: Quaternion

Relative rotation of the twin

Experimental

This API should not be used in production and may be trimmed from a public release.

---

scale?

protected optional scale: number

Scale of the twin

Experimental

This API should not be used in production and may be trimmed from a public release.

---

scene?

protected optional scene: Scene

Reference to a scene instance

Inherited from

PoseAlignPlugin.scene

---

skeleton?

protected optional skeleton: Skeleton

Reference to model’s skeleton

Inherited from

PoseAlignPlugin.skeleton

---

skeletonNodes?

protected optional skeletonNodes: SkeletonNodes

Bones of the model’s rig

Inherited from

PoseAlignPlugin.skeletonNodes

---

skeletonSizes?

protected optional skeletonSizes: SkeletonSizes

Reference sizes of skeleton

Inherited from

PoseAlignPlugin.skeletonSizes

---

spineCurve?

protected optional spineCurve: SpineCurve

Shape of spine

Inherited from

PoseAlignPlugin.spineCurve

---

translation?

protected optional translation: Vector3

Relative translation of the twin

Experimental

This API should not be used in production and may be trimmed from a public release.

---

tune

protected tune: PoseTuneParams

Fine-tuning parameters

Inherited from

PoseAlignPlugin.tune

Methods

dispose()

dispose(): void

Dispose video plugin

Returns

void

Inherited from

PoseAlignPlugin.dispose

---

estimateBones()

protected estimateBones(pose, spineCurve, skeletonSizes): SkeletonTransforms

Estimate bone positions and axes

Based on detected keypoints estimates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated from relative positions of adjacent keypoints. Method returns only bone position and orientation axis, final transformation of any bone can be found using the next bone in hierarchy.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• pose: PosePoints

Pose keypoints

• spineCurve: SpineCurve

Shape of spine

• skeletonSizes: SkeletonSizes

Returns

SkeletonTransforms

Bone transformations

Inherited from

PoseAlignPlugin.estimateBones

---

load()

load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• renderer: Renderer<PoseResult>

Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Inherited from

PoseAlignPlugin.load

---

setNode()

setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

Parameters

• node?: TransformNode

Scene node to attach

Returns

void

Inherited from

PoseAlignPlugin.setNode

---

setRotation()

setRotation(rotation?): void

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• rotation?: Quaternion

Returns

void

---

setScale()

setScale(scale?): void

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• scale?: number

Returns

void

---

setTranslation()

setTranslation(translation?): void

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• translation?: Vector3

Returns

void

---

setupCamera()

setupCamera(ratio, angle): void

Set camera parameters

Parameters

• ratio: number

Aspect ration of input video

• angle: number

Vertical field of view in radians

Returns

void

Inherited from

PoseAlignPlugin.setupCamera

---

setupVideo()

setupVideo(size): void

Set video size

Could be overridden to adjust plugin’s pipeline.

Parameters

• size: Size

Resolution of input video

Returns

void

Inherited from

PoseAlignPlugin.setupVideo

---

unload()

unload(): void

Reset plugin

Clears internal state and frees all resources allocated in load().

Returns

void

Inherited from

PoseAlignPlugin.unload

---

update()

update(result, stream): Promise<void>

Update skeleton of the scene node

Method is extended to set twin’s translation, rotation, and scale relative to estimated pose.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• result: PoseResult

Pose estimation results

• stream: HTMLCanvasElement

Captured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

PoseAlignPlugin.update

---

updateArm()

protected updateArm(bones, points): void

Update arm rig (see update)

Overridden to only rotate bones ignoring positions and relative scalings.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• bones

Arm bones and transforms

• bones.arm

• bones.arm.bone: TransformNode

• bones.arm.transform: BoneTransform

• bones.forearm

• bones.forearm.bone: TransformNode

• bones.forearm.transform: BoneTransform

• bones.hand

• bones.hand.bone: TransformNode

• bones.hand.transform: BoneTransform

• points

Arm keypoints

• points.elbow: PosePoint

• points.shoulder: PosePoint

Returns

void

Overrides

PoseAlignPlugin.updateArm

---

updateLeg()

protected updateLeg(bones, points): void

Update leg rig (see update)

Overridden to only rotate bones ignoring positions and relative scalings.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• bones

Leg bones and transforms

• bones.foot

• bones.foot.bone: TransformNode

• bones.foot.transform: BoneTransform

• bones.leg

• bones.leg.bone: TransformNode

• bones.leg.transform: BoneTransform

• bones.toe

• bones.toe.bone: TransformNode

• bones.toe.transform: BoneTransform

• bones.upleg

• bones.upleg.bone: TransformNode

• bones.upleg.transform: BoneTransform

• points

Leg keypoints

• points.ankle: PosePoint

• points.hip: PosePoint

• points.knee: PosePoint

Returns

void

Overrides

PoseAlignPlugin.updateLeg

---

updateSpine()

protected updateSpine(transforms, bones): void

Update spine rig (see update)

Overridden to only rotate bones ignoring positions and relative scalings.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• transforms: SkeletonTransforms

Estimated transformations

• bones: SkeletonNodes

Spine bones

Returns

void

Overrides

PoseAlignPlugin.updateSpine