Class: ClothAlignPlugin

@geenee/bodyrenderers-babylon.ClothAlignPlugin

Pose plugin aligning node's rig with keypoints

Universal plugin aligning node's rig and pose estimated by @geenee/bodyprocessors!PoseProcessor. This is the best starting point for advanced virtual try-on of apparel. Basically, ClothAlignPlugin evaluates positions and rotations of armature bones based on 3D pose keypoints, then applies these transforms to bones following the armature hierarchy. To improve accuracy of alignment, detected 3D points and skeleton sizes are mutually adjusted to fit each other. Plugin supports rigs compatible with Clo3D and Marvelous Designer avatars. This is the most common standard of rigs in cloth/apparel modeling software. Controlled scene node must contain an armature among its children nodes. Bones of armature must follow Clo3D naming convention and hierarchy. ClothAlignPlugin can apply number of fine-tuning adjustments to basic alignment improving model fitting or making it look more natural. PoseTuneParams explains tuning options.

Hierarchy

• PosePlugin

  ↳ ClothAlignPlugin

  ↳↳ ClothSkirtPlugin

Constructors

constructor

• new ClothAlignPlugin(node?, tune?)

Constructor

Parameters

Name	Type	Description
node?	TransformNode	Scene node to attach
tune	PoseTuneParams	Fine-tuning parameters

Overrides

PosePlugin.constructor

Properties

alignScore

• Readonly alignScore: 0.9

Pose score threshold

---

alignVisibility

• Readonly alignVisibility: 0.9

Keypoint visibility threshold

---

cameraAngle

• Protected cameraAngle: number

Camera vertical angle in radians

---

cameraRatio

• Protected cameraRatio: number

Camera aspect ratio

---

loaded

• loaded: boolean

Loaded state

Inherited from

PosePlugin.loaded

---

node

• Protected Optional node: TransformNode

Scene node to attach

---

nodeOrigin

• Protected nodeOrigin: BoneTransform

Origin of node relative to the root bone

---

ordinal

• ordinal: number

Ordinal number

Inherited from

PosePlugin.ordinal

---

renderer

• Protected Optional renderer: Renderer<PoseResult>

Renderer loaded the plugin

Inherited from

PosePlugin.renderer

---

scene

• Protected Optional scene: Scene

Reference to a scene instance

Inherited from

PosePlugin.scene

---

skeleton

• Protected Optional skeleton: Skeleton

Reference to model's skeleton

---

skeletonNodes

• Protected Optional skeletonNodes: SkeletonNodes

Bones of the model's rig

---

skeletonSizes

• Protected Optional skeletonSizes: SkeletonSizes

Reference sizes of skeleton

---

spineCurve

• Protected Optional spineCurve: SpineCurve

Shape of spine

---

tune

• Protected tune: PoseTuneParams

Fine-tuning parameters

Methods

dispose

▸ dispose(): void

Dispose video plugin

Returns

void

Inherited from

PosePlugin.dispose

---

estimateBones

▸ Protected estimateBones(pose, spineCurve, skeletonSizes): Object

Estimate bone positions and orientations

Using detected keypoints approximates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone's rotation is defined by its axes that are evaluated using kinematic rules and relative positions of adjacent keypoints. Method returns bone position and orientation in global world frame, final relative transformations are found traversing skeleton hierarchy and accumulating transforms of parents.

Parameters

Name	Type	Description
pose	PosePoints	Pose keypoints
spineCurve	SpineCurve	Reference shape of spine
skeletonSizes	SkeletonSizes	Reference sizes of skeleton

Returns

Object

Bone transformations

Name	Type
armL	BoneTransform
armMidL	BoneTransform
armMidR	BoneTransform
armR	BoneTransform
footL	BoneTransform
footR	BoneTransform
forearmL	BoneTransform
forearmMidL	BoneTransform
forearmMidR	BoneTransform
forearmR	BoneTransform
handL	BoneTransform
handR	BoneTransform
head	BoneTransform
legL	BoneTransform
legR	BoneTransform
neck	BoneTransform
neck1	BoneTransform
pelvis	BoneTransform
root	BoneTransform
shoulderL	BoneTransform
shoulderR	BoneTransform
spine	BoneTransform
spine1	BoneTransform
spine2	BoneTransform
spine3	BoneTransform
uplegL	BoneTransform
uplegMidL	BoneTransform
uplegMidR	BoneTransform
uplegR	BoneTransform

---

load

▸ load(renderer): Promise<void>

Initialize plugin

Parses and caches the rig/armature of the attached scene node (one provided to plugin's constructor). Precalculates geometrical parameters of skeleton.

Parameters

Name	Type	Description
renderer	Renderer<PoseResult>	Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Overrides

PosePlugin.load

---

setNode

▸ setNode(node?): void

Set/attach a scene node

Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.

Parameters

Name	Type	Description
node?	TransformNode	Scene node to attach

Returns

void

---

setupCamera

▸ setupCamera(ratio, angle): void

Set camera parameters

Parameters

Name	Type	Description
ratio	number	Aspect ration of input video
angle	number	Vertical field of view in radians

Returns

void

Overrides

PosePlugin.setupCamera

---

setupVideo

▸ setupVideo(size): void

Set video size

Could be overridden to adjust plugin's pipeline.

Parameters

Name	Type	Description
size	Size	Resolution of input video

Returns

void

Inherited from

PosePlugin.setupVideo

---

unload

▸ unload(): void

Reset plugin

Removes the attached node.

Returns

void

Overrides

PosePlugin.unload

---

update

▸ update(result, stream): Promise<void>

Update skeleton of the scene node

Evaluates positions, rotations and scales of node bones based on estimation of 3D keypoints, then applies these transformations to bones following hierarchy of armature. To improve accuracy of alignment detected 3D points and skeleton sizes are mutually adjusted to fit each other. One can override this method to further tune model's rig using provided estimations of bones as starting point. Simply call await super.update(result, stream); and use PoseAlignPlugin#skeletonNodes member storing refs to all bones of the skeleton to access transformations.

Parameters

Name	Type	Description
result	PoseResult	Pose estimation results
stream	HTMLCanvasElement	Captured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

PosePlugin.update