Face Tracking

FaceProcessor estimates 3D face landmarks, it detects and tracks face mesh providing smooth, stable and accurate results. Processor evaluates 2D pixel and 3D metric points as well as face pose (translation+rotation+scale) aligning reference face model. 2D pixel face landmarks - points in the screen coordinate space. X and Y coordinates are normalized screen coordinates (scaled by width and height of the input image), while the Z coordinate is depth within orthographic projection space. These points can be used for 2D face filters or when using orthographic projection. 3D metric points - points within 3D space of perspective camera located at the space origin and pointed in the negative direction of the Z-axis. These points can be used to apply texture face mask. 3D and 2D points are perfectly aligned, projections of 3D points coincide with 2D pixel coordinates within the perspective camera. Face pose - transformation matrix (translation+rotation+scale) aligning reference face 3D model with the measured 3D face mesh. Applying this transformation one can align 3D object with the detected face. If the model’s initial position is aligned with the reference face, relative transformation will be preserved.

Face processor may estimate an accurate & stable segmentation mask. Segmentation mask - monochrome image, where every pixel has value in range [0..1] denoting the probability of it being a foreground. Mask is provided for normalized rect region of the original image, it has a fixed size in pixels and should be scaled to image space. Optional temporal smoothing of a segmentation mask may be enabled. Estimated mask may be used for background substitution, effects like bokeh or focal blur, advanced occluder materials utilizing a mask, regional patchers, and other foreground/background shader effects.

FaceProcessor emits FaceResult storing results of face tracking that are passed to Renderer. When you create and setup application Engine for PoseProcessor, you need to provide FaceParams to enable or disable optional results of face tracking. Disabling an unused estimation can improve application performance. FaceEngine is a straightforward specialization of Engine for FaceProcessor.

Simple application utilizing FaceProcessor. In this application we use Snapshot helper to take a picture on click event. Additionally we will add occluder using a plugin.

import { FaceEngine } from "@geenee/bodyprocessors";
import { Snapshoter } from "@geenee/armature";
import { CustomRenderer } from "./customrenderer";
import "./index.css";

const engine = new FaceEngine();
const token = location.hostname === "localhost" ?
    "localhost_sdk_token" : "prod.url_sdk_token";

async function main() {
    const container = document.getElementById("root");
    if (!container)
        return;
    const renderer = new CustomRenderer(container);

    const snapshoter = new Snapshoter(renderer);
    container.onclick = async () => {
        const image = await snapshoter.snapshot();
        if (!image)
            return;
        const canvas = document.createElement("canvas");
        const context = canvas.getContext('2d');
        if (!context)
            return;
        canvas.width = image.width;
        canvas.height = image.height;
        context.putImageData(image, 0, 0);
        const url = canvas.toDataURL();
        const link = document.createElement("a");
        link.hidden = true;
        link.href = url;
        link.download = "capture.png";
        link.click();
        link.remove();
        URL.revokeObjectURL(url);
    }

    await Promise.all([
        engine.addRenderer(renderer),
        engine.init({ token: token, transform: true })]);
    await engine.setup({ size: { width: 1920, height: 1080 });
    await engine.start();
    document.getElementById("dots")?.remove();
}
main();

Documentation of the following packages provides details on how to build more extensive application with custom logic: