ClothTwinPlugin

Plugin controlling a digital twin

ClothAlignPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin’s scene node can be further transformed relative to the initial position - centers of hips are the same. Additionally, callback method can be assigned, it will be called after pose of the twin is set up, hips centers are aligned, and fixed transformations (translation, rotation, scale) are applied. Callback provides more flexible control over twin’s relative pose. Under construction, not well tested after changes.

Experimental

This API should not be used in production and may be trimmed from a public release.

Extends

• ClothAlignPlugin

Constructors

new ClothTwinPlugin()

new ClothTwinPlugin(node?, translation?, rotation?, scale?, callback?, tune?): ClothTwinPlugin

Constructor

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• node?: TransformNode

Scene node to attach

• translation?: Vector3

Relative translation of the twin

• rotation?: Quaternion

Relative rotation of the twin

• scale?: number

Scale of the twin

• callback?

Method called after pose is set up

• tune?: PoseTuneParams = ...

Fine-tuning parameters

Returns

ClothTwinPlugin

Overrides

ClothAlignPlugin.constructor

Properties

alignScore

readonly alignScore: 0.9 = 0.90

Pose score threshold

Inherited from

ClothAlignPlugin.alignScore

---

alignVisibility

readonly alignVisibility: 0.9 = 0.90

Keypoint visibility threshold

Inherited from

ClothAlignPlugin.alignVisibility

---

callback()?

protected optional callback: (node) => void

Method called after pose is set up

Parameters

• node: TransformNode

Returns

void

---

cameraAngle

protected cameraAngle: number

Camera vertical angle in radians

Inherited from

ClothAlignPlugin.cameraAngle

---

cameraRatio

protected cameraRatio: number

Camera aspect ratio

Inherited from

ClothAlignPlugin.cameraRatio

---

loaded

loaded: boolean

Loaded state

Inherited from

ClothAlignPlugin.loaded

---

node?

protected optional node: TransformNode

Scene node to attach

Inherited from

ClothAlignPlugin.node

---

nodeOrigin

protected nodeOrigin: BoneTransform

Origin of node relative to the root bone

Inherited from

ClothAlignPlugin.nodeOrigin

---

ordinal

ordinal: number

Ordinal number

Inherited from

ClothAlignPlugin.ordinal

---

renderer?

protected optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

Inherited from

ClothAlignPlugin.renderer

---

rotation?

protected optional rotation: Quaternion

Relative rotation of the twin

Experimental

This API should not be used in production and may be trimmed from a public release.

---

scale?

protected optional scale: number

Scale of the twin

Experimental

This API should not be used in production and may be trimmed from a public release.

---

scene?

protected optional scene: Scene

Reference to a scene instance

Inherited from

ClothAlignPlugin.scene

---

skeleton?

protected optional skeleton: Skeleton

Reference to model’s skeleton

Inherited from

ClothAlignPlugin.skeleton

---

skeletonNodes?

protected optional skeletonNodes: SkeletonNodes

Bones of the model’s rig

Inherited from

ClothAlignPlugin.skeletonNodes

---

skeletonSizes?

protected optional skeletonSizes: SkeletonSizes

Reference sizes of skeleton

Inherited from

ClothAlignPlugin.skeletonSizes

---

spineCurve?

protected optional spineCurve: SpineCurve

Shape of spine

Inherited from

ClothAlignPlugin.spineCurve

---

translation?

protected optional translation: Vector3

Relative translation of the twin

Experimental

This API should not be used in production and may be trimmed from a public release.

---

tune

protected tune: PoseTuneParams

Fine-tuning parameters

Inherited from

ClothAlignPlugin.tune

Methods

dispose()

dispose(): void

Dispose video plugin

Returns

void

Inherited from

ClothAlignPlugin.dispose

---

estimateBones()

protected estimateBones(pose, spineCurve, skeletonSizes): object

Estimate bone positions and orientations

Using detected keypoints approximates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated using kinematic rules and relative positions of adjacent keypoints. Method returns bone position and orientation in global world frame, final relative transformations are found traversing skeleton hierarchy and accumulating transforms of parents.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• pose: PosePoints

Pose keypoints

• spineCurve: SpineCurve

Reference shape of spine

• skeletonSizes: SkeletonSizes

Reference sizes of skeleton

Returns

object

Bone transformations

armL

armL: BoneTransform = armL.arm

armMidL

armMidL: BoneTransform = armL.armMid

armMidR

armMidR: BoneTransform = armR.armMid

armR

armR: BoneTransform = armR.arm

footL

footL: BoneTransform = legL.foot

footR

footR: BoneTransform = legR.foot

forearmL

forearmL: BoneTransform = armL.forearm

forearmMidL

forearmMidL: BoneTransform = armL.forearmMid

forearmMidR

forearmMidR: BoneTransform = armR.forearmMid

forearmR

forearmR: BoneTransform = armR.forearm

handL

handL: BoneTransform = armL.hand

handR

handR: BoneTransform = armR.hand

head

head: BoneTransform

legL

legL: BoneTransform = legL.leg

legR

legR: BoneTransform = legR.leg

neck

neck: BoneTransform

neck1

neck1: BoneTransform

pelvis

pelvis: BoneTransform

root

root: BoneTransform

shoulderL

shoulderL: BoneTransform = armL.shoulder

shoulderR

shoulderR: BoneTransform = armR.shoulder

spine

spine: BoneTransform

spine1

spine1: BoneTransform

spine2

spine2: BoneTransform

spine3

spine3: BoneTransform

uplegL

uplegL: BoneTransform = legL.upleg

uplegMidL

uplegMidL: BoneTransform = legL.uplegMid

uplegMidR

uplegMidR: BoneTransform = legR.uplegMid

uplegR

uplegR: BoneTransform = legR.upleg

Inherited from

ClothAlignPlugin.estimateBones

---

load()

load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• renderer: Renderer<PoseResult>

Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Inherited from

ClothAlignPlugin.load

---

setCallback()

setCallback(callback): void

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• callback: undefined | (node) => void

Returns

void

---

setNode()

setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

Parameters

• node?: TransformNode

Scene node to attach

Returns

void

Inherited from

ClothAlignPlugin.setNode

---

setRotation()

setRotation(rotation?): void

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• rotation?: Quaternion

Returns

void

---

setScale()

setScale(scale?): void

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• scale?: number

Returns

void

---

setTranslation()

setTranslation(translation?): void

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• translation?: Vector3

Returns

void

---

setupCamera()

setupCamera(ratio, angle): void

Set camera parameters

Parameters

• ratio: number

Aspect ration of input video

• angle: number

Vertical field of view in radians

Returns

void

Inherited from

ClothAlignPlugin.setupCamera

---

setupVideo()

setupVideo(size): void

Set video size

Could be overridden to adjust plugin’s pipeline.

Parameters

• size: Size

Resolution of input video

Returns

void

Inherited from

ClothAlignPlugin.setupVideo

---

unload()

unload(): void

Reset plugin

Clears internal state and frees all resources allocated in load().

Returns

void

Inherited from

ClothAlignPlugin.unload

---

update()

update(result, stream): Promise<void>

Update skeleton of the scene node

Evaluates positions, rotations and scales of node bones based on estimation of 3D keypoints, then applies these transformations to bones following hierarchy of armature. To improve accuracy of alignment detected 3D points and skeleton sizes are mutually adjusted to fit each other. One can override this method to further tune model’s rig using provided estimations of bones as a starting point. Simply call await super.update(result, stream); and use PoseAlignPlugin#skeletonNodes member storing refs to all bones of the skeleton to access transformations.

Experimental

This API should not be used in production and may be trimmed from a public release.

Parameters

• result: PoseResult

Pose estimation results

• stream: HTMLCanvasElement

Captured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

ClothAlignPlugin.update

---

updateArm()

protected updateArm(bones, points): void

Update arm rig (see update)

Parameters

• bones

Arm bones and transforms

• bones.arm

• bones.arm.bone: TransformNode

• bones.arm.transform: BoneTransform

• bones.armMid

• bones.armMid.bone: TransformNode

• bones.armMid.transform: BoneTransform

• bones.forearm

• bones.forearm.bone: TransformNode

• bones.forearm.transform: BoneTransform

• bones.forearmMid

• bones.forearmMid.bone: TransformNode

• bones.forearmMid.transform: BoneTransform

• bones.hand

• bones.hand.bone: TransformNode

• bones.hand.transform: BoneTransform

• points

Arm keypoints

• points.elbow: PosePoint

• points.shoulder: PosePoint

Returns

void

Overrides

ClothAlignPlugin.updateArm

---

updateLeg()

protected updateLeg(bones, points): void

Update leg rig (see update)

Parameters

• bones

Leg bones and transforms

• bones.foot

• bones.foot.bone: TransformNode

• bones.foot.transform: BoneTransform

• bones.leg

• bones.leg.bone: TransformNode

• bones.leg.transform: BoneTransform

• bones.upleg

• bones.upleg.bone: TransformNode

• bones.upleg.transform: BoneTransform

• bones.uplegMid

• bones.uplegMid.bone: TransformNode

• bones.uplegMid.transform: BoneTransform

• points

Leg keypoints

• points.hip: PosePoint

• points.knee: PosePoint

Returns

void

Overrides

ClothAlignPlugin.updateLeg

---

updateSpine()

protected updateSpine(transforms, bones): void

Update spine rig (see update)

Parameters

• transforms

Estimated transformations

• transforms.armL: BoneTransform = armL.arm

• transforms.armMidL: BoneTransform = armL.armMid

• transforms.armMidR: BoneTransform = armR.armMid

• transforms.armR: BoneTransform = armR.arm

• transforms.footL: BoneTransform = legL.foot

• transforms.footR: BoneTransform = legR.foot

• transforms.forearmL: BoneTransform = armL.forearm

• transforms.forearmMidL: BoneTransform = armL.forearmMid

• transforms.forearmMidR: BoneTransform = armR.forearmMid

• transforms.forearmR: BoneTransform = armR.forearm

• transforms.handL: BoneTransform = armL.hand

• transforms.handR: BoneTransform = armR.hand

• transforms.head: BoneTransform = ...

• transforms.legL: BoneTransform = legL.leg

• transforms.legR: BoneTransform = legR.leg

• transforms.neck: BoneTransform = ...

• transforms.neck1: BoneTransform = ...

• transforms.pelvis: BoneTransform = ...

• transforms.root: BoneTransform = ...

• transforms.shoulderL: BoneTransform = armL.shoulder

• transforms.shoulderR: BoneTransform = armR.shoulder

• transforms.spine: BoneTransform = ...

• transforms.spine1: BoneTransform = ...

• transforms.spine2: BoneTransform = ...

• transforms.spine3: BoneTransform = ...

• transforms.uplegL: BoneTransform = legL.upleg

• transforms.uplegMidL: BoneTransform = legL.uplegMid

• transforms.uplegMidR: BoneTransform = legR.uplegMid

• transforms.uplegR: BoneTransform = legR.upleg

• bones: SkeletonNodes

Spine bones

Returns

void

Overrides

ClothAlignPlugin.updateSpine