ClothAlignPlugin
Pose plugin aligning node’s rig with keypoints
Universal plugin aligning node’s rig and pose estimated by @geenee/bodyprocessors!PoseProcessor. This is the best starting point for advanced virtual try-on of apparel. Basically, ClothAlignPlugin evaluates positions and rotations of armature bones based on 3D pose keypoints, then applies these transforms to bones following the armature hierarchy. To improve accuracy of alignment, detected 3D points and skeleton sizes are mutually adjusted to fit each other. Plugin supports rigs compatible with Clo3D and Marvelous Designer avatars. This is the most common standard of rigs in cloth/apparel modeling software. Controlled scene node must contain an armature among its children nodes. Bones of armature must follow Clo3D naming convention and hierarchy. ClothAlignPlugin can apply number of fine-tuning adjustments to basic alignment improving model fitting or making it look more natural. PoseTuneParams explains tuning options.
Extends
Extended by
Constructors
new ClothAlignPlugin()
new ClothAlignPlugin(
node?,tune?):ClothAlignPlugin
Constructor
Parameters
• node?: TransformNode
Scene node to attach
• tune?: PoseTuneParams = ...
Fine-tuning parameters
Returns
Overrides
Properties
alignScore
readonlyalignScore:0.9=0.90
Pose score threshold
alignVisibility
readonlyalignVisibility:0.9=0.90
Keypoint visibility threshold
cameraAngle
protectedcameraAngle:number
Camera vertical angle in radians
cameraRatio
protectedcameraRatio:number
Camera aspect ratio
loaded
loaded:
boolean
Loaded state
Inherited from
node?
protectedoptionalnode:TransformNode
Scene node to attach
nodeOrigin
protectednodeOrigin:BoneTransform
Origin of node relative to the root bone
ordinal
ordinal:
number
Ordinal number
Inherited from
renderer?
protectedoptionalrenderer:Renderer<PoseResult>
Renderer loaded the plugin
Inherited from
scene?
protectedoptionalscene:Scene
Reference to a scene instance
Inherited from
skeleton?
protectedoptionalskeleton:Skeleton
Reference to model’s skeleton
skeletonNodes?
protectedoptionalskeletonNodes:SkeletonNodes
Bones of the model’s rig
skeletonSizes?
protectedoptionalskeletonSizes:SkeletonSizes
Reference sizes of skeleton
spineCurve?
protectedoptionalspineCurve:SpineCurve
Shape of spine
tune
protectedtune:PoseTuneParams
Fine-tuning parameters
Methods
dispose()
dispose():
void
Dispose video plugin
Returns
void
Inherited from
estimateBones()
protectedestimateBones(pose,spineCurve,skeletonSizes):object
Estimate bone positions and orientations
Using detected keypoints approximates bone transformations. Position of bone if defined by 3D point itself, bone length is the distance between keypoints connected by bone. Bone’s rotation is defined by its axes that are evaluated using kinematic rules and relative positions of adjacent keypoints. Method returns bone position and orientation in global world frame, final relative transformations are found traversing skeleton hierarchy and accumulating transforms of parents.
Parameters
• pose: PosePoints
Pose keypoints
• spineCurve: SpineCurve
Reference shape of spine
• skeletonSizes: SkeletonSizes
Reference sizes of skeleton
Returns
object
Bone transformations
armL
armL:
BoneTransform=armL.arm
armMidL
armMidL:
BoneTransform=armL.armMid
armMidR
armMidR:
BoneTransform=armR.armMid
armR
armR:
BoneTransform=armR.arm
footL
footL:
BoneTransform=legL.foot
footR
footR:
BoneTransform=legR.foot
forearmL
forearmL:
BoneTransform=armL.forearm
forearmMidL
forearmMidL:
BoneTransform=armL.forearmMid
forearmMidR
forearmMidR:
BoneTransform=armR.forearmMid
forearmR
forearmR:
BoneTransform=armR.forearm
handL
handL:
BoneTransform=armL.hand
handR
handR:
BoneTransform=armR.hand
head
head:
BoneTransform
legL
legL:
BoneTransform=legL.leg
legR
legR:
BoneTransform=legR.leg
neck
neck:
BoneTransform
neck1
neck1:
BoneTransform
pelvis
pelvis:
BoneTransform
root
root:
BoneTransform
shoulderL
shoulderL:
BoneTransform=armL.shoulder
shoulderR
shoulderR:
BoneTransform=armR.shoulder
spine
spine:
BoneTransform
spine1
spine1:
BoneTransform
spine2
spine2:
BoneTransform
spine3
spine3:
BoneTransform
uplegL
uplegL:
BoneTransform=legL.upleg
uplegMidL
uplegMidL:
BoneTransform=legL.uplegMid
uplegMidR
uplegMidR:
BoneTransform=legR.uplegMid
uplegR
uplegR:
BoneTransform=legR.upleg
load()
load(
renderer):Promise<void>
Initialize plugin
Parses and caches the rig/armature of the attached scene node (one provided to plugin’s constructor). Precalculates geometrical parameters of skeleton.
Parameters
• renderer: Renderer<PoseResult>
Renderer this plugin is attached to
Returns
Promise<void>
Promise resolving when initialization is finished
Overrides
setNode()
setNode(
node?):void
Set/attach a scene node
Parses and caches the rig/armature of the node. Precalculates geometrical parameters of skeleton.
Parameters
• node?: TransformNode
Scene node to attach
Returns
void
setupCamera()
setupCamera(
ratio,angle):void
Set camera parameters
Parameters
• ratio: number
Aspect ration of input video
• angle: number
Vertical field of view in radians
Returns
void
Overrides
setupVideo()
setupVideo(
size):void
Set video size
Could be overridden to adjust plugin’s pipeline.
Parameters
• size: Size
Resolution of input video
Returns
void
Inherited from
unload()
unload():
void
Reset plugin
Clears internal state and frees all resources allocated in load().
Returns
void
Overrides
update()
update(
result,stream):Promise<void>
Update skeleton of the scene node
Evaluates positions, rotations and scales of node bones
based on estimation of 3D keypoints, then applies these
transformations to bones following hierarchy of armature.
To improve accuracy of alignment detected 3D points and
skeleton sizes are mutually adjusted to fit each other.
One can override this method to further tune model’s rig
using provided estimations of bones as a starting point.
Simply call await super.update(result, stream); and use
PoseAlignPlugin#skeletonNodes member storing refs
to all bones of the skeleton to access transformations.
Parameters
• result: PoseResult
Pose estimation results
• stream: HTMLCanvasElement
Captured video frame
Returns
Promise<void>
Promise resolving when update is finished
Overrides
updateArm()
protectedupdateArm(bones,points):void
Update arm rig (see update)
Parameters
• bones
Arm bones and transforms
• bones.arm
• bones.arm.bone: TransformNode
• bones.arm.transform: BoneTransform
• bones.armMid
• bones.armMid.bone: TransformNode
• bones.armMid.transform: BoneTransform
• bones.forearm
• bones.forearm.bone: TransformNode
• bones.forearm.transform: BoneTransform
• bones.forearmMid
• bones.forearmMid.bone: TransformNode
• bones.forearmMid.transform: BoneTransform
• bones.hand
• bones.hand.bone: TransformNode
• bones.hand.transform: BoneTransform
• points
Arm keypoints
• points.elbow: PosePoint
• points.shoulder: PosePoint
Returns
void
updateLeg()
protectedupdateLeg(bones,points):void
Update leg rig (see update)
Parameters
• bones
Leg bones and transforms
• bones.foot
• bones.foot.bone: TransformNode
• bones.foot.transform: BoneTransform
• bones.leg
• bones.leg.bone: TransformNode
• bones.leg.transform: BoneTransform
• bones.upleg
• bones.upleg.bone: TransformNode
• bones.upleg.transform: BoneTransform
• bones.uplegMid
• bones.uplegMid.bone: TransformNode
• bones.uplegMid.transform: BoneTransform
• points
Leg keypoints
• points.hip: PosePoint
• points.knee: PosePoint
Returns
void
updateSpine()
protectedupdateSpine(transforms,bones):void
Update spine rig (see update)
Parameters
• transforms
Estimated transformations
• transforms.armL: BoneTransform = armL.arm
• transforms.armMidL: BoneTransform = armL.armMid
• transforms.armMidR: BoneTransform = armR.armMid
• transforms.armR: BoneTransform = armR.arm
• transforms.footL: BoneTransform = legL.foot
• transforms.footR: BoneTransform = legR.foot
• transforms.forearmL: BoneTransform = armL.forearm
• transforms.forearmMidL: BoneTransform = armL.forearmMid
• transforms.forearmMidR: BoneTransform = armR.forearmMid
• transforms.forearmR: BoneTransform = armR.forearm
• transforms.handL: BoneTransform = armL.hand
• transforms.handR: BoneTransform = armR.hand
• transforms.head: BoneTransform = ...
• transforms.legL: BoneTransform = legL.leg
• transforms.legR: BoneTransform = legR.leg
• transforms.neck: BoneTransform = ...
• transforms.neck1: BoneTransform = ...
• transforms.pelvis: BoneTransform = ...
• transforms.root: BoneTransform = ...
• transforms.shoulderL: BoneTransform = armL.shoulder
• transforms.shoulderR: BoneTransform = armR.shoulder
• transforms.spine: BoneTransform = ...
• transforms.spine1: BoneTransform = ...
• transforms.spine2: BoneTransform = ...
• transforms.spine3: BoneTransform = ...
• transforms.uplegL: BoneTransform = legL.upleg
• transforms.uplegMidL: BoneTransform = legL.uplegMid
• transforms.uplegMidR: BoneTransform = legR.uplegMid
• transforms.uplegR: BoneTransform = legR.upleg
• bones: SkeletonNodes
Spine bones
Returns
void