GSoC 2017 First Phase Summary

2017/06/27

Summary of progress of implementing face tracking mechanism in Jitsi-Meet, in GSoC 2017.

Overview

Jitsi-Meet supports secure, simple and scalable video conferences which could be used as a standalone app or embed in a web application. In GSoC (Google Summer of Code) 2017, I was accepted by Jitsi organization to develop a face tracking mechanism for Jitsi-Meet, under the guide of Yana Stamcheva and Saúl Ibarra Corretgé.

By June 27th, I have added basic face tracking mechanism in:

  • preview video in device setting dialog. The tracking functionality here has been enabled by default and cannot be disabled by users.

  • large video in video container. The tracking mechanism here could be enabled/disabled by the checkbox at the settings menu.

For the tracking, to be specific, if a user's face is not well-positioned (in the center area of the screen) for a while (6s), necessary prompts including a rectangle representing good position and a text prompt with warning message, will be displayed to ask the user to move to the center.

I list related results generated from my current work:

Setup of Development Environment

The development environment is mainly deployed in my local machine, 13'' MacBook Air 2014. Because of the usage of Webpack in Jitsi-Meet, I setup a proxy server as the back-end server. And the concrete configuration is shown below.

/**
 * The URL of the Jitsi Meet deployment to be proxy to in the context of
 * development with webpack-dev-server.
 */
const devServerProxyTarget
    = process.env.WEBPACK_DEV_SERVER_PROXY_TARGET || 'https://beta.meet.jit.si';

Additionally, Jitsi org helps me to fire a virtual server which holds https://chunxu.jitsi.net. In this case, for every release, I can deploy it onto the virtual server for essential review and discussion.

Face Tracking in Jitsi-Meet

tracking.js library

tracking.js is a JavaScript library which brings different computer vision algorithms and techniques into the browser environment. I use the library as the backbone of face tracking mechanism in Jitsi-Meet. While, even though the library has an NPM module, the module has broken, and cannot be used by ES6 code. Additionally, I also need to add new features such as limiting frames per second. As a result, I forked the official repository, and made necessary modifications (my repo).

FaceTracker and FacePrompt

In my design, FaceTracker is a React component responsible for face tracking with tracking.js library, and FacePrompt is another component taking charge of rendering UI components. And preview video and large video component call FaceTracker respectively.

The image below shows an example of the relationship between preview video, FaceTracker and FacePrompt.

Image of components

To be specific, when necessary, VideoInputPreview component triggers enableFaceTracking() action, which is handled by corresponding reducer(s), and modifies the state in Redux store. The change will be reflected on FaceTracker via _mapStateToProps() function. After the tracking is enabled, if the user's face is not well-positioned for a while, FacePrompt will be created via render() function of FaceTracker.

Demo of current work

Image of large video Face tracking in the large video

Image of large video Face tracking in the preview video

Future Work

The todos below are listed in descending priority order.

  1. Fix bugs in current implementation. The current face tracking mechanism still exists some bugs. For example, Yana figures out that when the device setting dialog open, closed, and then open again, the preview video will disappear (bug link). In the following month, I need to fix the bugs in the implementation.

  2. Improve the structure of code. My current design of face tracking is far from perfect, and there's still much space to enhance. For example, Saúl has helped me to split face tracking mechanism into two modules: one for face tracking, and the other for UI rendering (link). Additionally, Yana recommended me to change FaceTracker React component to a middleware module (link).

  3. Integrate the face tracking mechanism with other functionalities. Currently, I haven't cared much about the integration with other modules. For example, I need to auto-disable face tracking in the large video when it’s muted. I also plan to auto-resize face prompts when full screen mode is toggled.

  4. Rewrite tracking.js library. My current work involves in adding new features in original tracking.js library, mainly in build directory. While, this is not a good coding practice to directly inject new code there. The problem is that tracking.js library doesn't support CommonJS or ES6 modules, which makes it difficult to add new functionality. I plan to rewrite the library in ES6 modules, as well as adding new necessary features.

Post Directory