PoseFitPlugin

Pose plugin aligning node’s rig with keypoints

Universal plugin aligning node’s rig and pose estimated by @geenee/bodyprocessors!PoseProcessor. This is the best starting point for advanced virtual try-on of apparel. Basically, PoseFitPlugin evaluates positions and rotations of armature bones based on 3D pose keypoints, then applies these transforms to bones following the armature hierarchy. To improve accuracy of alignment, detected 3D points and skeleton sizes are mutually adjusted to fit each other. Plugin supports rigs compatible with Clo3D and Marvelous Designer avatars. This is the most common standard of rigs in cloth/apparel modeling software. Controlled scene node must contain an armature among its children nodes. Bones of armature must follow Clo3D naming convention and hierarchy. PoseFitPlugin can apply number of fine-tuning adjustments to basic alignment improving model fitting or making it look more natural. PoseTuneParams explains tuning options.

Extends

PosePlugin

Extended by

Constructors

Constructor

new PoseFitPlugin(tune): PoseFitPlugin

Constructor

Parameters

tune

PoseTuneParams = ...

Fine-tuning parameters

Returns

PoseFitPlugin

Overrides

PosePlugin.constructor

Properties

alignScore

readonly alignScore: 0.9 = 0.90

Pose score threshold

alignVisibility

readonly alignVisibility: 0.9 = 0.90

Keypoint visibility threshold

cameraAngle

protected cameraAngle: number

Camera vertical angle in radians

cameraRatio

protected cameraRatio: number

Camera aspect ratio

loaded

loaded: boolean

Loaded state

Inherited from

PosePlugin.loaded

node?

protected optional node: TransformNode

Attached scene node

nodeOrigin

protected nodeOrigin: BoneTransform

Origin of node relative to the root bone

ordinal

ordinal: number

Ordinal number

Inherited from

PosePlugin.ordinal

renderer?

protected optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

Inherited from

PosePlugin.renderer

scene?

protected optional scene: Scene

Reference to a scene instance

Inherited from

PosePlugin.scene

skeleton?

protected optional skeleton: Skeleton

Reference to model’s skeleton

skeletonNodes?

protected optional skeletonNodes: SkeletonNodes

Bones of the model’s rig

skeletonSizes?

protected optional skeletonSizes: SkeletonSizes

Reference sizes of skeleton

spineCurve?

protected optional spineCurve: SpineCurve

Shape of spine

tune

protected tune: PoseTuneParams

Fine-tuning parameters

Methods

dispose()

dispose(): void

Dispose video plugin

Returns

void

Inherited from

PosePlugin.dispose

estimateBones()

protected estimateBones(pose, spineCurve, skeletonSizes): object

Estimate bone positions and orientations

Using detected keypoints approximates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated using kinematic rules and relative positions of adjacent keypoints. Method returns bone position and orientation in global world frame, final relative transformations are found traversing skeleton hierarchy and accumulating transforms of parents.

Parameters

pose

PosePoints

Pose keypoints

spineCurve

SpineCurve

Reference shape of spine

skeletonSizes

SkeletonSizes

Reference sizes of skeleton

Returns

object

Bone transformations

armL

armL: BoneTransform = armL.arm

armMidL

armMidL: BoneTransform = armL.armMid

armMidR

armMidR: BoneTransform = armR.armMid

armR

armR: BoneTransform = armR.arm

footL

footL: BoneTransform = legL.foot

footR

footR: BoneTransform = legR.foot

forearmL

forearmL: BoneTransform = armL.forearm

forearmMidL

forearmMidL: BoneTransform = armL.forearmMid

forearmMidR

forearmMidR: BoneTransform = armR.forearmMid

forearmR

forearmR: BoneTransform = armR.forearm

handL

handL: BoneTransform = armL.hand

handR

handR: BoneTransform = armR.hand

head

head: BoneTransform

legL

legL: BoneTransform = legL.leg

legR

legR: BoneTransform = legR.leg

neck

neck: BoneTransform

neck1

neck1: BoneTransform

pelvis

pelvis: BoneTransform

root

root: BoneTransform

shoulderL

shoulderL: BoneTransform = armL.shoulder

shoulderR

shoulderR: BoneTransform = armR.shoulder

spine

spine: BoneTransform

spine1

spine1: BoneTransform

spine2

spine2: BoneTransform

spine3

spine3: BoneTransform

uplegL

uplegL: BoneTransform = legL.upleg

uplegMidL

uplegMidL: BoneTransform = legL.uplegMid

uplegMidR

uplegMidR: BoneTransform = legR.uplegMid

uplegR

uplegR: BoneTransform = legR.upleg

load()

load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.

Parameters

renderer

Renderer<PoseResult>

Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Overrides

PosePlugin.load

setNode()

setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

Parameters

node?

TransformNode

Scene node to attach

Returns

void

setupCamera()

setupCamera(ratio, angle): void

Set camera parameters

Parameters

ratio

number

Aspect ration of input video

angle

number

Vertical field of view in radians

Returns

void

Overrides

PosePlugin.setupCamera

setupVideo()

setupVideo(size): void

Set video size

Could be overridden to adjust plugin’s pipeline.

Parameters

size

Size

Resolution of input video

Returns

void

Inherited from

PosePlugin.setupVideo

unload()

unload(): void

Reset plugin

Clears internal state and frees all resources allocated in load().

Returns

void

Overrides

PosePlugin.unload

update()

update(result, stream): Promise<void>

Update skeleton of the scene node

Evaluates positions, rotations and scales of node bones based on estimation of 3D keypoints, then applies these transformations to bones following hierarchy of armature. To improve accuracy of alignment detected 3D points and skeleton sizes are mutually adjusted to fit each other. One can override this method to further tune model’s rig using provided estimations of bones as a starting point. Simply call await super.update(result, stream); and use PoseFitPlugin#skeletonNodes member storing refs to all bones of the skeleton to access transformations.

Parameters

result

PoseResult

Pose estimation results

stream

HTMLCanvasElement

Captured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

PosePlugin.update

updateArm()

protected updateArm(bones, points): void

Update arm rig (see update)

void

updateLeg()

protected updateLeg(bones, points): void

Update leg rig (see update)

PosePoint

Returns

void

updateSpine()

protected updateSpine(transforms, bones): void

Update spine rig (see update)

BoneTransform = legR.upleg

bones

SkeletonNodes

Spine bones

Returns

void