PoseBFitPlugin

Pose plugin aligning node’s rig with keypoints

Universal plugin aligning node’s rig and pose estimated by @geenee/bodyprocessors!PoseProcessor. It’s a base of try-on, twin, and other plugins. You can use this class as a starting point and customize alignment method or add features. Basically, PoseBFitPlugin evaluates positions and rotations of armature bones based on 3D pose keypoints, then applies these transforms to bones following the armature hierarchy. Plugin supports rigs compatible with Mixamo, for example any Ready Player Me avatar. This is the most common standard of rigs for human-like models supported by many game engines. Provided node must contain an armature among its children. Armature bones must follow Mixamo / RPM naming convention. Models rigged and skinned manually or using Mixamo tool can variate depending on anthropomorphous topology of the model. PoseBFitPlugin can apply number of fine-tuning adjustments to basic alignment improving model fitting or making it look more natural. PoseTuneParams explains tuning options. By default the plugin is fine-tuned for RPM avatars so you can simply replace person with the avatar model in the scene.

Extends

PosePlugin

Extended by

PoseBTwinPlugin

Constructors

Constructor

new PoseBFitPlugin(tune): PoseBFitPlugin

Constructor

Parameters

tune

PoseTuneParams = ...

Fine-tuning parameters

Returns

PoseBFitPlugin

Overrides

PosePlugin.constructor

Properties

alignScore

readonly alignScore: 0.9 = 0.90

Pose score threshold

alignVisibility

readonly alignVisibility: 0.9 = 0.90

Keypoint visibility threshold

avatarLength

protected avatarLength: number = 1

Reference length of the model

loaded

loaded: boolean

Loaded state

Inherited from

PosePlugin.loaded

node?

protected optional node: Object3D<Object3DEventMap>

Attached scene node

ordinal

ordinal: number

Ordinal number

Inherited from

PosePlugin.ordinal

renderer?

protected optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

Inherited from

PosePlugin.renderer

scene?

protected optional scene: Scene

Reference to a scene instance

Inherited from

PosePlugin.scene

skeleton?

protected optional skeleton: Skeleton

Reference to model’s skeleton

skeletonNodes?

protected optional skeletonNodes: SkeletonNodes

Bones of the model’s rig

spineCurve?

protected optional spineCurve: SpineCurve

Shape of spine

tune

protected tune: PoseTuneParams

Fine-tuning parameters

Methods

alignBone()

protected alignBone(transform, bone, scale): void

Set bone transformation

Transformation is in the world coordinate frame. If bone has parent we find relative transformation to follow the skeleton hierarchy. Optionally scale of parent bone can be adjusted to align head and reference (rest) position of parent’s tail (they have to be the same 3D point).

Parameters

transform

BoneTransform

Global bone position and rotation

bone

Bone

Reference to bone instance

scale

boolean = true

Whether to scale parent bone

Returns

void

dispose()

dispose(): void

Dispose video plugin

Returns

void

Inherited from

PosePlugin.dispose

estimateBones()

protected estimateBones(points, spineCurve): SkeletonTransforms

Estimate bone positions and axes

Based on detected keypoints estimates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated from relative positions of adjacent keypoints. Method returns only bone position and orientation axis, final transformation of any bone can be found using the next bone in hierarchy.

Parameters

points

PosePoints

Pose keypoints

spineCurve

SpineCurve

Shape of spine

Returns

SkeletonTransforms

Bone transformations

load()

load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.

Parameters

renderer

Renderer<PoseResult>

Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Overrides

PosePlugin.load

setNode()

setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

Parameters

node?

Object3D<Object3DEventMap>

Scene node to attach

Returns

void

setupCamera()

setupCamera(ratio, angle): void

Set camera parameters

Could be overridden to adjust plugin’s pipeline.

Parameters

ratio

number

Aspect ration of input video

angle

number

Vertical field of view in radians

Returns

void

Inherited from

PosePlugin.setupCamera

setupVideo()

setupVideo(size): void

Set video size

Could be overridden to adjust plugin’s pipeline.

Parameters

size

Size

Resolution of input video

Returns

void

Inherited from

PosePlugin.setupVideo

unload()

unload(): void

Reset plugin

Removes the attached node.

Returns

void

Overrides

PosePlugin.unload

update()

update(result, stream): Promise<void>

Update skeleton of the scene node

Evaluates positions, rotations and scales of node bones based on estimation of 3D keypoints, then applies these transformations to bones following hierarchy of armature. Optionally fine-tunes the basic alignment to improve model fitting or make it more natural. You can override this method to further tune model’s rig using provided estimations of bones as starting point. Simply call await super.update(result, stream); and use PoseBFitPlugin#skeletonNodes member storing refs to all bones of the skeleton to access transformations.