Class: PoseTwinPlugin
@geenee/bodyrenderers-babylon.PoseTwinPlugin
Plugin rendering a digital twin
PoseAlignPlugin extension for digital twins mirroring the pose and residing beside a user. When rendering a twin we do not translate bones to align with keypoint coordinates and only preserve relative rotations. After projecting the detected pose onto a twin, twin's scene node can be further transformed relative to the initial position - centers of hips are the same.
Hierarchy
↳
PoseTwinPlugin
Constructors
constructor
• new PoseTwinPlugin(node?
, translation?
, rotation?
, scale?
, tune?
)
Constructor
Parameters
Name | Type | Description |
---|---|---|
node? | TransformNode | Scene node to attach |
translation? | Vector3 | Relative translation of the twin |
rotation? | Quaternion | Relative rotation of the twin |
scale? | number | Scale of the twin |
tune | PoseTuneParams | Fine-tuning parameters |
Overrides
Properties
alignScore
• Readonly
alignScore: 0.9
Pose score threshold
Inherited from
alignVisibility
• Readonly
alignVisibility: 0.9
Keypoint visibility threshold
Inherited from
PoseAlignPlugin.alignVisibility
avatarLength
• Protected
avatarLength: number
= 1
Reference length of the model
Inherited from
loaded
• loaded: boolean
Loaded state
Inherited from
node
• Protected
Optional
node: TransformNode
Inherited from
renderer
• Protected
Optional
renderer: Renderer
<PoseResult
>
Renderer loaded the plugin
Inherited from
rotation
• Protected
Optional
rotation: Quaternion
scale
• Protected
Optional
scale: number
scene
• Protected
Optional
scene: Scene
Reference to a scene instance
Inherited from
skeleton
• Protected
Optional
skeleton: SkeletonRef
Bones of the model's rig
Inherited from
skeletonRef
• Protected
Optional
skeletonRef: Skeleton
Reference to model's skeleton
Inherited from
spineRef
• Protected
Optional
spineRef: Spine
Shape of spine
Inherited from
translation
• Protected
Optional
translation: Vector3
tune
• Protected
tune: PoseTuneParams
Inherited from
Methods
alignBone
▸ Protected
alignBone(transform
, bone
): void
Set bone aligning transformation
Overridden to assign only relative rotation.
Parameters
Name | Type | Description |
---|---|---|
transform | BoneTransform | Global bone position and rotation |
bone | TransformNode | Reference to bone instance |
Returns
void
Overrides
dispose
▸ dispose(): void
Dispose video plugin
Returns
void
Inherited from
estimateBones
▸ Protected
estimateBones(points
, spineRef
): Skeleton
Estimate bone positions and axes
Based on detected keypoints estimates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone's rotation is defined by its axes that are evaluated from relative positions of adjacent keypoints. Method returns only bone position and orientation axis, final transformation of any bone can be found using the next bone in hierarchy.
Parameters
Name | Type | Description |
---|---|---|
points | PosePoints | Pose keypoints |
spineRef | Spine | Shape of spine |
Returns
Bone transformations
Inherited from
load
▸ load(renderer
): Promise
<void
>
Initialize plugin
Parses and caches the rig/armature of the attached scene node (one provided to plugin's constructor). Precalculates geometrical parameters of skeleton.
Parameters
Name | Type | Description |
---|---|---|
renderer | Renderer <PoseResult > | Renderer this plugin is attached to |
Returns
Promise
<void
>
Promise resolving when initialization is finished
Inherited from
setNode
▸ setNode(node?
): void
Set/attach a scene node
Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.
Parameters
Name | Type | Description |
---|---|---|
node? | TransformNode | Scene node to attach |
Returns
void
Promise resolving when initialization is finished
Inherited from
setupVideo
▸ setupVideo(size
): void
Set video size
Could be overridden to adjust plugin's pipeline.
Parameters
Name | Type | Description |
---|---|---|
size | Size | Resolution of input video |
Returns
void
Inherited from
unload
▸ unload(): void
Reset plugin
Removes the attached node.
Returns
void
Inherited from
update
▸ update(result
, stream
): Promise
<void
>
Update skeleton of the scene node
Method is extended to set twin's translation, rotation, and scale relative to estimated pose.
Parameters
Name | Type | Description |
---|---|---|
result | PoseResult | Pose estimation results |
stream | HTMLCanvasElement | Captured video frame |
Returns
Promise
<void
>
Promise resolving when update is finished
Overrides
updateHandL
▸ Protected
updateHandL(anchors
, points
): void
Update left hand skeleton
Parameters
Name | Type | Description |
---|---|---|
anchors | Skeleton | Positions and axes of bones |
points | PosePoints | Pose keypoints |
Returns
void
Inherited from
updateHandR
▸ Protected
updateHandR(anchors
, points
): void
Update right hand skeleton
Parameters
Name | Type | Description |
---|---|---|
anchors | Skeleton | Positions and axes of bones |
points | PosePoints | Pose keypoints |
Returns
void
Inherited from
updateLegL
▸ Protected
updateLegL(anchors
, points
): void
Update left leg skeleton
Parameters
Name | Type | Description |
---|---|---|
anchors | Skeleton | Positions and axes of bones |
points | PosePoints | Pose keypoints |
Returns
void
Inherited from
updateLegR
▸ Protected
updateLegR(anchors
, points
): void
Update right leg skeleton
Parameters
Name | Type | Description |
---|---|---|
anchors | Skeleton | Positions and axes of bones |
points | PosePoints | Pose keypoints |
Returns
void
Inherited from
updateSpine
▸ Protected
updateSpine(anchors
): void
Update spine skeleton
Overridden to ignore relative scaling.
Parameters
Name | Type | Description |
---|---|---|
anchors | Skeleton | Positions and axes of bones |
Returns
void