Body Tracking Models

Body Tracking Models

Occluders

Occluders are visual elements that represent real objects in a scene and are used to hide parts of AR objects behind them. They are not rendered themselves but participate in depth test, in other words they only signal not to draw parts of meshes behind them being completely transparent at the same time. For body tracking AR, especially try-on, body occluder helps to hide parts of apparel behind user’s real body. This basic model of a body can be used as occluder in body tracking applications.

Usage:

• Use OccluderPlugin to make a scene node be an occluder.
• Use PoseAlignPlugin to control occluder’s pose.

Models

This section will show how to prepare 3D models for apps utilizing body tracking. This type of tracking estimates the position and rotation of joints on the body. The most common use case it to align armature of a skinned 3D model with pose of the body - avatar overlay and virtual apparel try-on AR experiences.

Basics

Our SDK supports the next structure of armature/skeleton:

Hips
  Spine
    Spine1
      Spine2
        Neck
          Head
            HeadTop_End
      LeftShoulder
        LeftArm
          LeftForeArm
            LeftHand
              ...
      RightShoulder
        RightArm
          RightForeArm
            RightHand
              ...
  LeftUpLeg
    LeftLeg
      LeftFoot
        LeftToeBase
          ...
  RightUpLeg
    RightLeg
      RightFoot
        RightToeBase
          ...

You can download reference armature here. Any model with this skeleton should be compatible with our body tracking. It is required that names of bones are as described. One exception - you can use any prefixes in bone names, like for example “mixamo:LeftUpLeg” or “prefix_LeftArm”. Parsing of the skeleton is case insensitive, for example, LeftArm bone may have any name ending with “LeftArm”, “leftArm”, or “leftarm”.

This armature/skeleton is standard and default for game engines (Godot, Unreal, Unity), tools (Mixamo, Lens Studio), avatar systems (Ready Player Me), etc. For example any Ready Player Me avatar is compatible with our body tracking out-of-the-box. You can also use these avatars as starting point for your own character or outfit model.

In general, preparation of models for body tracking is the same as any other 3D model and you can use any 3D tool you prefer: Blender, Maya, Cinema4D, etc. The main difference is that final model should be skinned. This process requires some skill, but no worries - it’s not so difficult. There’re a lot of materials on how to do it. In the next section we’ll describe several simple methods of skinning a model.

Skinning

Skinning is the process of binding the actual 3D mesh to the joint setup (armature). This means that the joints you setup will have influence on the vertices of your model and move them accordingly. We will describe several simple approaches to skin a model.

• Automatic weights
• Weight transfer
• Mixamo

Usage

• Use PoseAlignPlugin to align rig of a scene node and estimated pose. Node’s armature will follow the pose and overlay on top of the real body.
• Use PoseOutfitPlugin to specify body meshes of avatar’s scene node as occluders and optionally hide some child meshes for virtual try-on.
• Use PoseTwinPlugin to make a rigged scene node be a digital twin mirroring the pose and residing beside a user.
• Example for babylon.js
• Example for three.js