HandFitPlugin
Hand plugin fitting geometry into keypoints
Universal plugin fitting HandGeometry into a pose estimated by @geenee/bodyprocessors!HandProcessor. HandFitPlugin evaluates positions, rotations and scales of phalanxes based on detected keypoints, and then applies these transforms to sub-components of a hand geometry. Use HandFitPlugin#buildGeometry to construct geometry.
Extends
Constructors
new HandFitPlugin()
new HandFitPlugin(
geometry?):HandFitPlugin
Constructor
Parameters
• geometry?: HandGeometry
Hand geometry to attach
Returns
Overrides
Properties
geometry?
protectedoptionalgeometry:HandGeometry
Hand geometry to attach
loaded
loaded:
boolean
Loaded state
Inherited from
ordinal
ordinal:
number
Ordinal number
Inherited from
renderer?
protectedoptionalrenderer:Renderer<HandResult>
Renderer loaded the plugin
Inherited from
scene?
protectedoptionalscene:Scene
Reference to a scene instance
Inherited from
Methods
dispose()
dispose():
void
Dispose video plugin
Returns
void
Inherited from
estimateBones()
protectedestimateBones(pose,handedness):object
Estimate phalanxes positions and orientations
Using detected keypoints approximates transformations of phalanxes. Position of a phalanx is defined by 3D point of the corresponding joint. Its length is the distance between keypoints defining the phalanx. Rotation is defined by its axes that are evaluated using kinematic rules and relative positions of adjacent joint keypoints. Method returns phalanx position & orientation in global world frame.
Parameters
• pose: HandPoint[]
Pose keypoints
• handedness: number
Returns
object
Phalanx transformations
hand
hand:
BoneTransform
index
index:
BoneTransform[]
middle
middle:
BoneTransform[]
pinky
pinky:
BoneTransform[]
ring
ring:
BoneTransform[]
thumb
thumb:
BoneTransform[]
load()
load(
renderer):Promise<void>
Initialize plugin
Prepares or modifies the attached node if required. Reference to the scene object is cached and used by plugin on update() and unload(). You need to reload plugin if you want to change scene it’s attached to.
Parameters
• renderer: Renderer<HandResult>
Renderer this plugin is attached to
Returns
Promise<void>
Promise resolving when initialization is finished
Inherited from
setGeometry()
setGeometry(
geometry?):void
Set/attach a hand geometry
Parameters
• geometry?: HandGeometry
Hand geometry to attach
Returns
void
setupCamera()
setupCamera(
ratio,angle):void
Set camera parameters
Could be overridden to adjust plugin’s pipeline.
Parameters
• ratio: number
Aspect ration of input video
• angle: number
Vertical field of view in radians
Returns
void
Inherited from
setupVideo()
setupVideo(
size):void
Set video size
Could be overridden to adjust plugin’s pipeline.
Parameters
• size: Size
Resolution of input video
Returns
void
Inherited from
unload()
unload():
void
Reset plugin
Releases all resources allocated in load(). Deletes cached reference to the scene object.
Returns
void
Inherited from
update()
update(
result,stream):Promise<void>
Update pose of the hand geometry
Evaluates positions, rotations and scales of pahalanxes
based on 3D keypoints and other data, then applies these
transformations to components of the attached geometry.
One can override this method to further tune geometry of
a hand using default imlpementation as a starting point.
Simply call await super.update(result, stream); first
and adjust poses assigned to components of hand geometry.
Parameters
• result: HandResult
Hand estimation results
• stream: HTMLCanvasElement
Captured video frame
Returns
Promise<void>
Promise resolving when update is finished
Overrides
buildGeometry()
staticbuildGeometry(scene,material):HandGeometry
Build a hand geometry
Hand geometry includes 4 fingers (index, middle, ring, pinky). Each finger consists of 3 cylinders for phalanxes and 4 spheres for joints beetween them and on their open ends. Phalanxes and spheres have unit radiuses and lengthes for easier computations.
Parameters
• scene: Scene
Scene to add geometry to
• material: Material
Material of the mesh
Returns
Hand geometry