PoseTwinPlugin
Plugin controlling a digital twin
PoseFitPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin’s scene node can be further transformed relative to the initial position - centers of hips are the same. Additionally, callback method can be assigned, it will be called after pose of the twin is set up, hips centers are aligned, and fixed transformations (translation, rotation, scale) are applied. Callback provides more flexible control over twin’s relative pose. Under construction, not well tested after changes.
Extends
Section titled “Extends”Constructors
Section titled “Constructors”Constructor
Section titled “Constructor”new PoseTwinPlugin(
translation?,rotation?,scale?,callback?,tune?):PoseTwinPlugin
Constructor
Parameters
Section titled “Parameters”translation?
Section titled “translation?”Vector3
Relative translation of the twin
rotation?
Section titled “rotation?”Quaternion
Relative rotation of the twin
scale?
Section titled “scale?”number
Scale of the twin
callback?
Section titled “callback?”(node) => void
Method called after pose is set up
Fine-tuning parameters
Returns
Section titled “Returns”PoseTwinPlugin
Overrides
Section titled “Overrides”Properties
Section titled “Properties”alignScore
Section titled “alignScore”
readonlyalignScore:0.9=0.90
Pose score threshold
Inherited from
Section titled “Inherited from”alignVisibility
Section titled “alignVisibility”
readonlyalignVisibility:0.9=0.90
Keypoint visibility threshold
Inherited from
Section titled “Inherited from”callback()?
Section titled “callback()?”
protectedoptionalcallback: (node) =>void
Method called after pose is set up
Parameters
Section titled “Parameters”TransformNode
Returns
Section titled “Returns”void
cameraAngle
Section titled “cameraAngle”
protectedcameraAngle:number
Camera vertical angle in radians
Inherited from
Section titled “Inherited from”cameraRatio
Section titled “cameraRatio”
protectedcameraRatio:number
Camera aspect ratio
Inherited from
Section titled “Inherited from”loaded
Section titled “loaded”loaded:
boolean
Loaded state
Inherited from
Section titled “Inherited from”
protectedoptionalnode:TransformNode
Attached scene node
Inherited from
Section titled “Inherited from”nodeOrigin
Section titled “nodeOrigin”
protectednodeOrigin:BoneTransform
Origin of node relative to the root bone
Inherited from
Section titled “Inherited from”ordinal
Section titled “ordinal”ordinal:
number
Ordinal number
Inherited from
Section titled “Inherited from”renderer?
Section titled “renderer?”
protectedoptionalrenderer:Renderer<PoseResult>
Renderer loaded the plugin
Inherited from
Section titled “Inherited from”rotation?
Section titled “rotation?”
protectedoptionalrotation:Quaternion
Relative rotation of the twin
scale?
Section titled “scale?”
protectedoptionalscale:number
Scale of the twin
scene?
Section titled “scene?”
protectedoptionalscene:Scene
Reference to a scene instance
Inherited from
Section titled “Inherited from”skeleton?
Section titled “skeleton?”
protectedoptionalskeleton:Skeleton
Reference to model’s skeleton
Inherited from
Section titled “Inherited from”skeletonNodes?
Section titled “skeletonNodes?”
protectedoptionalskeletonNodes:SkeletonNodes
Bones of the model’s rig
Inherited from
Section titled “Inherited from”skeletonSizes?
Section titled “skeletonSizes?”
protectedoptionalskeletonSizes:SkeletonSizes
Reference sizes of skeleton
Inherited from
Section titled “Inherited from”spineCurve?
Section titled “spineCurve?”
protectedoptionalspineCurve:SpineCurve
Shape of spine
Inherited from
Section titled “Inherited from”translation?
Section titled “translation?”
protectedoptionaltranslation:Vector3
Relative translation of the twin
protectedtune:PoseTuneParams
Fine-tuning parameters
Inherited from
Section titled “Inherited from”Methods
Section titled “Methods”dispose()
Section titled “dispose()”dispose():
void
Dispose video plugin
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”estimateBones()
Section titled “estimateBones()”
protectedestimateBones(pose,spineCurve,skeletonSizes):object
Estimate bone positions and orientations
Using detected keypoints approximates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated using kinematic rules and relative positions of adjacent keypoints. Method returns bone position and orientation in global world frame, final relative transformations are found traversing skeleton hierarchy and accumulating transforms of parents.
Parameters
Section titled “Parameters”Pose keypoints
spineCurve
Section titled “spineCurve”SpineCurve
Reference shape of spine
skeletonSizes
Section titled “skeletonSizes”SkeletonSizes
Reference sizes of skeleton
Returns
Section titled “Returns”object
Bone transformations
armL:
BoneTransform=armL.arm
armMidL
Section titled “armMidL”armMidL:
BoneTransform=armL.armMid
armMidR
Section titled “armMidR”armMidR:
BoneTransform=armR.armMid
armR:
BoneTransform=armR.arm
footL:
BoneTransform=legL.foot
footR:
BoneTransform=legR.foot
forearmL
Section titled “forearmL”forearmL:
BoneTransform=armL.forearm
forearmMidL
Section titled “forearmMidL”forearmMidL:
BoneTransform=armL.forearmMid
forearmMidR
Section titled “forearmMidR”forearmMidR:
BoneTransform=armR.forearmMid
forearmR
Section titled “forearmR”forearmR:
BoneTransform=armR.forearm
handL:
BoneTransform=armL.hand
handR:
BoneTransform=armR.hand
head:
BoneTransform
legL:
BoneTransform=legL.leg
legR:
BoneTransform=legR.leg
neck:
BoneTransform
neck1:
BoneTransform
pelvis
Section titled “pelvis”pelvis:
BoneTransform
root:
BoneTransform
shoulderL
Section titled “shoulderL”shoulderL:
BoneTransform=armL.shoulder
shoulderR
Section titled “shoulderR”shoulderR:
BoneTransform=armR.shoulder
spine:
BoneTransform
spine1
Section titled “spine1”spine1:
BoneTransform
spine2
Section titled “spine2”spine2:
BoneTransform
spine3
Section titled “spine3”spine3:
BoneTransform
uplegL
Section titled “uplegL”uplegL:
BoneTransform=legL.upleg
uplegMidL
Section titled “uplegMidL”uplegMidL:
BoneTransform=legL.uplegMid
uplegMidR
Section titled “uplegMidR”uplegMidR:
BoneTransform=legR.uplegMid
uplegR
Section titled “uplegR”uplegR:
BoneTransform=legR.upleg
Inherited from
Section titled “Inherited from”load()
Section titled “load()”load(
renderer):Promise<void>
Initialize plugin
Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.
Parameters
Section titled “Parameters”renderer
Section titled “renderer”Renderer this plugin is attached to
Returns
Section titled “Returns”Promise<void>
Promise resolving when initialization is finished
Inherited from
Section titled “Inherited from”setCallback()
Section titled “setCallback()”setCallback(
callback):void
Parameters
Section titled “Parameters”callback
Section titled “callback”undefined | (node) => void
Returns
Section titled “Returns”void
setNode()
Section titled “setNode()”setNode(
node?):void
Set/attach a scene node
Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.
Parameters
Section titled “Parameters”TransformNode
Scene node to attach
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”setRotation()
Section titled “setRotation()”setRotation(
rotation?):void
Parameters
Section titled “Parameters”rotation?
Section titled “rotation?”Quaternion
Returns
Section titled “Returns”void
setScale()
Section titled “setScale()”setScale(
scale?):void
Parameters
Section titled “Parameters”scale?
Section titled “scale?”number
Returns
Section titled “Returns”void
setTranslation()
Section titled “setTranslation()”setTranslation(
translation?):void
Parameters
Section titled “Parameters”translation?
Section titled “translation?”Vector3
Returns
Section titled “Returns”void
setupCamera()
Section titled “setupCamera()”setupCamera(
ratio,angle):void
Set camera parameters
Parameters
Section titled “Parameters”number
Aspect ration of input video
number
Vertical field of view in radians
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”setupVideo()
Section titled “setupVideo()”setupVideo(
size):void
Set video size
Could be overridden to adjust plugin’s pipeline.
Parameters
Section titled “Parameters”Resolution of input video
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”unload()
Section titled “unload()”unload():
void
Reset plugin
Clears internal state and frees all resources allocated in load().
Returns
Section titled “Returns”void
Inherited from
Section titled “Inherited from”update()
Section titled “update()”update(
result,stream):Promise<void>
Update skeleton of the scene node
Evaluates positions, rotations and scales of node bones
based on estimation of 3D keypoints, then applies these
transformations to bones following hierarchy of armature.
To improve accuracy of alignment detected 3D points and
skeleton sizes are mutually adjusted to fit each other.
One can override this method to further tune model’s rig
using provided estimations of bones as a starting point.
Simply call await super.update(result, stream); and use
PoseFitPlugin#skeletonNodes member storing refs
to all bones of the skeleton to access transformations.
Parameters
Section titled “Parameters”result
Section titled “result”Pose estimation results
stream
Section titled “stream”HTMLCanvasElement
Captured video frame
Returns
Section titled “Returns”Promise<void>
Promise resolving when update is finished
Overrides
Section titled “Overrides”updateArm()
Section titled “updateArm()”
protectedupdateArm(bones,points):void
Update arm rig (see update)
Parameters
Section titled “Parameters”Arm bones and transforms
{ bone: TransformNode; transform: BoneTransform; }
arm.bone
Section titled “arm.bone”TransformNode
arm.transform
Section titled “arm.transform”armMid
Section titled “armMid”{ bone: TransformNode; transform: BoneTransform; }
armMid.bone
Section titled “armMid.bone”TransformNode
armMid.transform
Section titled “armMid.transform”forearm
Section titled “forearm”{ bone: TransformNode; transform: BoneTransform; }
forearm.bone
Section titled “forearm.bone”TransformNode
forearm.transform
Section titled “forearm.transform”forearmMid
Section titled “forearmMid”{ bone: TransformNode; transform: BoneTransform; }
forearmMid.bone
Section titled “forearmMid.bone”TransformNode
forearmMid.transform
Section titled “forearmMid.transform”{ bone: TransformNode; transform: BoneTransform; }
hand.bone
Section titled “hand.bone”TransformNode
hand.transform
Section titled “hand.transform”points
Section titled “points”Arm keypoints
shoulder
Section titled “shoulder”Returns
Section titled “Returns”void
Overrides
Section titled “Overrides”updateLeg()
Section titled “updateLeg()”
protectedupdateLeg(bones,points):void
Update leg rig (see update)
Parameters
Section titled “Parameters”Leg bones and transforms
{ bone: TransformNode; transform: BoneTransform; }
foot.bone
Section titled “foot.bone”TransformNode
foot.transform
Section titled “foot.transform”{ bone: TransformNode; transform: BoneTransform; }
leg.bone
Section titled “leg.bone”TransformNode
leg.transform
Section titled “leg.transform”{ bone: TransformNode; transform: BoneTransform; }
upleg.bone
Section titled “upleg.bone”TransformNode
upleg.transform
Section titled “upleg.transform”uplegMid
Section titled “uplegMid”{ bone: TransformNode; transform: BoneTransform; }
uplegMid.bone
Section titled “uplegMid.bone”TransformNode
uplegMid.transform
Section titled “uplegMid.transform”points
Section titled “points”Leg keypoints
Returns
Section titled “Returns”void
Overrides
Section titled “Overrides”updateSpine()
Section titled “updateSpine()”
protectedupdateSpine(transforms,bones):void
Update spine rig (see update)
Parameters
Section titled “Parameters”transforms
Section titled “transforms”Estimated transformations
BoneTransform = armL.arm
armMidL
Section titled “armMidL”BoneTransform = armL.armMid
armMidR
Section titled “armMidR”BoneTransform = armR.armMid
BoneTransform = armR.arm
BoneTransform = legL.foot
BoneTransform = legR.foot
forearmL
Section titled “forearmL”BoneTransform = armL.forearm
forearmMidL
Section titled “forearmMidL”BoneTransform = armL.forearmMid
forearmMidR
Section titled “forearmMidR”BoneTransform = armR.forearmMid
forearmR
Section titled “forearmR”BoneTransform = armR.forearm
BoneTransform = armL.hand
BoneTransform = armR.hand
BoneTransform = ...
BoneTransform = legL.leg
BoneTransform = legR.leg
BoneTransform = ...
BoneTransform = ...
pelvis
Section titled “pelvis”BoneTransform = ...
BoneTransform = ...
shoulderL
Section titled “shoulderL”BoneTransform = armL.shoulder
shoulderR
Section titled “shoulderR”BoneTransform = armR.shoulder
BoneTransform = ...
spine1
Section titled “spine1”BoneTransform = ...
spine2
Section titled “spine2”BoneTransform = ...
spine3
Section titled “spine3”BoneTransform = ...
uplegL
Section titled “uplegL”BoneTransform = legL.upleg
uplegMidL
Section titled “uplegMidL”BoneTransform = legL.uplegMid
uplegMidR
Section titled “uplegMidR”BoneTransform = legR.uplegMid
uplegR
Section titled “uplegR”BoneTransform = legR.upleg
SkeletonNodes
Spine bones
Returns
Section titled “Returns”void