PoseAlignPlugin
Pose plugin aligning node’s rig with keypoints
Universal plugin aligning node’s rig and pose estimated by @geenee/bodyprocessors!PoseProcessor. It’s a base of try-on, twin, and other plugins. You can use this class as a starting point and customize alignment method or add features. Basically, PoseAlignPlugin evaluates positions and rotations of armature bones based on 3D pose keypoints, then applies these transforms to bones following the armature hierarchy. Plugin supports rigs compatible with Mixamo, for example any Ready Player Me avatar. This is the most common standard of rigs for human-like models supported by many game engines. Provided node must contain an armature among its children. Armature bones must follow Mixamo / RPM naming convention. Models rigged and skinned manually or using Mixamo tool can variate depending on anthropomorphous topology of the model. PoseAlignPlugin can apply number of fine-tuning adjustments to basic alignment improving model fitting or making it look more natural. PoseTuneParams explains tuning options. By default the plugin is fine-tuned for RPM avatars so you can simply replace person with the avatar model in the scene.
Extends
Extended by
Constructors
new PoseAlignPlugin()
new PoseAlignPlugin(
node?,tune?):PoseAlignPlugin
Constructor
Parameters
• node?: Object3D<Object3DEventMap>
Scene node to attach
• tune?: PoseTuneParams = ...
Fine-tuning parameters
Returns
Overrides
Properties
alignScore
readonlyalignScore:0.9=0.90
Pose score threshold
alignVisibility
readonlyalignVisibility:0.9=0.90
Keypoint visibility threshold
avatarLength
protectedavatarLength:number=1
Reference length of the model
loaded
loaded:
boolean
Loaded state
Inherited from
node?
protectedoptionalnode:Object3D<Object3DEventMap>
Scene node to attach
ordinal
ordinal:
number
Ordinal number
Inherited from
renderer?
protectedoptionalrenderer:Renderer<PoseResult>
Renderer loaded the plugin
Inherited from
scene?
protectedoptionalscene:Scene
Reference to a scene instance
Inherited from
skeleton?
protectedoptionalskeleton:Skeleton
Reference to model’s skeleton
skeletonNodes?
protectedoptionalskeletonNodes:SkeletonNodes
Bones of the model’s rig
spineCurve?
protectedoptionalspineCurve:SpineCurve
Shape of spine
tune
protectedtune:PoseTuneParams
Fine-tuning parameters
Methods
alignBone()
protectedalignBone(transform,bone,scale):void
Set bone transformation
Transformation is in the world coordinate frame. If bone has parent we find relative transformation to follow the skeleton hierarchy. Optionally scale of parent bone can be adjusted to align head and reference (rest) position of parent’s tail (they have to be the same 3D point).
Parameters
• transform: BoneTransform
Global bone position and rotation
• bone: Bone<Object3DEventMap>
Reference to bone instance
• scale: boolean = true
Whether to scale parent bone
Returns
void
dispose()
dispose():
void
Dispose video plugin
Returns
void
Inherited from
estimateBones()
protectedestimateBones(points,spineCurve):SkeletonTransforms
Estimate bone positions and axes
Based on detected keypoints estimates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated from relative positions of adjacent keypoints. Method returns only bone position and orientation axis, final transformation of any bone can be found using the next bone in hierarchy.
Parameters
• points: PosePoints
Pose keypoints
• spineCurve: SpineCurve
Shape of spine
Returns
Bone transformations
load()
load(
renderer):Promise<void>
Initialize plugin
Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.
Parameters
• renderer: Renderer<PoseResult>
Renderer this plugin is attached to
Returns
Promise<void>
Promise resolving when initialization is finished
Overrides
setNode()
setNode(
node?):void
Set/attach a scene node
Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.
Parameters
• node?: Object3D<Object3DEventMap>
Scene node to attach
Returns
void
setupCamera()
setupCamera(
ratio,angle):void
Set camera parameters
Could be overridden to adjust plugin’s pipeline.
Parameters
• ratio: number
Aspect ration of input video
• angle: number
Vertical field of view in radians
Returns
void
Inherited from
setupVideo()
setupVideo(
size):void
Set video size
Could be overridden to adjust plugin’s pipeline.
Parameters
• size: Size
Resolution of input video
Returns
void
Inherited from
unload()
unload():
void
Reset plugin
Removes the attached node.
Returns
void
Overrides
update()
update(
result,stream):Promise<void>
Update skeleton of the scene node
Evaluates positions, rotations and scales of node bones
based on estimation of 3D keypoints, then applies these
transformations to bones following hierarchy of armature.
Optionally fine-tunes the basic
alignment to improve model fitting or make it more natural.
You can override this method to further tune model’s rig
using provided estimations of bones as starting point.
Simply call await super.update(result, stream); and use
PoseAlignPlugin#skeletonNodes member storing refs
to all bones of the skeleton to access transformations.
Parameters
• result: PoseResult
Pose estimation results
• stream: HTMLCanvasElement
Captured video frame
Returns
Promise<void>
Promise resolving when update is finished
Overrides
updateHandL()
protectedupdateHandL(anchors,points):void
Update left hand skeleton
Parameters
• anchors: SkeletonTransforms
Positions and axes of bones
• points: PosePoints
Pose keypoints
Returns
void
updateHandR()
protectedupdateHandR(anchors,points):void
Update right hand skeleton
Parameters
• anchors: SkeletonTransforms
Positions and axes of bones
• points: PosePoints
Pose keypoints
Returns
void
updateLegL()
protectedupdateLegL(anchors,points):void
Update left leg skeleton
Parameters
• anchors: SkeletonTransforms
Positions and axes of bones
• points: PosePoints
Pose keypoints
Returns
void
updateLegR()
protectedupdateLegR(anchors,points):void
Update right leg skeleton
Parameters
• anchors: SkeletonTransforms
Positions and axes of bones
• points: PosePoints
Pose keypoints
Returns
void
updateSpine()
protectedupdateSpine(anchors):void
Update spine skeleton
Parameters
• anchors: SkeletonTransforms
Positions and axes of bones
Returns
void