Skip to main content

Class: WristTrackPlugin

@geenee/bodyrenderers-babylon.WristTrackPlugin

Plugin assigning wrist pose to a scene node

Plugin attaches provided scene node to the wrist. Pose of the node (translation + rotation + scale) continuously updates according to pose estimation by @geenee/bodyprocessors!HandProcessor. Children nodes inherently include this transform. The node can be seen as a virtual placeholder for real object. It's recommended to attach top-level nodes that don't include transforms relative to parent, otherwise wrist transform that is a pose in the world frame will be applied on top of them (will be treated as relative instead of absolute). If wrist/hand isn't detected plugin hides the node. One of approaches to accurately align meshes with a wrist pose when modeling a scene is to make them children of one node at the origin and set their relative transforms using the wrist base mesh as the reference, then instantiate WristTrackPlugin for this scene node. You can also apply relative transforms of children of the wrist-attached parent node programmatically. It's useful to add occluder model (base mesh of a wrist) as a child of the node. Another possible but less scalable approach is to have all meshes be built relative to the origin and aligned with the base mesh of whe wrist, in this case you can create WristTrackPlugin for each mesh. This can be handy when parts are stored separately.

Hierarchy

Constructors

constructor

new WristTrackPlugin(node?)

Constructor

Parameters

NameTypeDescription
node?TransformNodeScene node to attach

Overrides

BabylonPlugin.constructor

Properties

cameraAngle

Protected cameraAngle: number

Camera vertical angle in radians


cameraRatio

Protected cameraRatio: number

Camera aspect ratio


loaded

loaded: boolean

Loaded state

Inherited from

BabylonPlugin.loaded


node

Protected Optional node: TransformNode

Scene node to attach


ordinal

ordinal: number

Ordinal number

Inherited from

BabylonPlugin.ordinal


renderer

Protected Optional renderer: Renderer<HandResult>

Renderer loaded the plugin

Inherited from

BabylonPlugin.renderer


scene

Protected Optional scene: Scene

Reference to a scene instance

Inherited from

BabylonPlugin.scene

Methods

dispose

dispose(): void

Dispose video plugin

Returns

void

Inherited from

BabylonPlugin.dispose


load

load(renderer): Promise<void>

Initialize plugin

Prepares or modifies the attached node if required. Reference to the scene object is cached and used by plugin on update() and unload(). You need to reload plugin if you want to change scene it's attached to.

Parameters

NameTypeDescription
rendererRenderer<HandResult>Renderer this plugin is attached to

Returns

Promise<void>

Promise resolving when initialization is finished

Inherited from

BabylonPlugin.load


setNode

setNode(node?): void

Set/attach a scene node

Parameters

NameTypeDescription
node?TransformNodeScene node to attach

Returns

void


setupCamera

setupCamera(ratio, angle): void

Set camera parameters

Parameters

NameTypeDescription
rationumberAspect ration of input video
anglenumberVertical field of view in radians

Returns

void

Overrides

BabylonPlugin.setupCamera


setupVideo

setupVideo(size): void

Set video size

Could be overridden to adjust plugin's pipeline.

Parameters

NameTypeDescription
sizeSizeResolution of input video

Returns

void

Inherited from

BabylonPlugin.setupVideo


unload

unload(): void

Reset plugin

Releases all resources allocated in load(). Deletes cached reference to the scene object.

Returns

void

Inherited from

BabylonPlugin.unload


update

update(result, stream): Promise<void>

Update pose of the node

Updates node's transform (translation+rotation+scale) according to the wrist estimation by @geenee/bodyprocessors!HandProcessor. If wrist isn't detected plugin recursively hides the attached node.

Parameters

NameTypeDescription
resultHandResultResults of video processing
streamHTMLCanvasElementCaptured video frame

Returns

Promise<void>

Promise resolving when update is finished

Overrides

BabylonPlugin.update