Summary of progress of implementing face tracking mechanism in Jitsi-Meet, in GSoC 2017.
Jitsi-Meet supports secure, simple and scalable video conferences which could be used as a standalone app or embed in a web application. In GSoC (Google Summer of Code) 2017, I was accepted by Jitsi organization to develop a face tracking mechanism for Jitsi-Meet, under the guide of Yana Stamcheva and Saúl Ibarra Corretgé.
By June 27th, I have added basic face tracking mechanism in:
preview video in device setting dialog. The tracking functionality here has been enabled by default and cannot be disabled by users.
large video in video container. The tracking mechanism here could be enabled/disabled by the checkbox at the settings menu.
For the tracking, to be specific, if a user's face is not well-positioned (in the center area of the screen) for a while (6s), necessary prompts including a rectangle representing
good position and a text prompt with warning message, will be displayed to ask the user to move to the center.
I list related results generated from my current work:
report of my progress in the first phase - this blog.
Setup of Development Environment
The development environment is mainly deployed in my local machine, 13'' MacBook Air 2014. Because of the usage of Webpack in Jitsi-Meet, I setup a proxy server as the back-end server. And the concrete configuration is shown below.
Additionally, Jitsi org helps me to fire a virtual server which holds https://chunxu.jitsi.net. In this case, for every release, I can deploy it onto the virtual server for essential review and discussion.
Face Tracking in Jitsi-Meet
FaceTracker and FacePrompt
In my design,
FaceTracker is a React component responsible for face tracking with tracking.js library, and
FacePrompt is another component taking charge of rendering UI components. And preview video and large video component call FaceTracker respectively.
The image below shows an example of the relationship between preview video,
To be specific, when necessary,
VideoInputPreview component triggers
enableFaceTracking() action, which is handled by corresponding reducer(s), and modifies the state in Redux store. The change will be reflected on
_mapStateToProps() function. After the tracking is enabled, if the user's face is not well-positioned for a while,
FacePrompt will be created via
render() function of
Demo of current work
Face tracking in the large video
Face tracking in the preview video
The todos below are listed in descending priority order.
Fix bugs in current implementation. The current face tracking mechanism still exists some bugs. For example, Yana figures out that when the device setting dialog open, closed, and then open again, the preview video will disappear (bug link). In the following month, I need to fix the bugs in the implementation.
Improve the structure of code. My current design of face tracking is far from perfect, and there's still much space to enhance. For example, Saúl has helped me to split face tracking mechanism into two modules: one for face tracking, and the other for UI rendering (link). Additionally, Yana recommended me to change FaceTracker React component to a middleware module (link).
Integrate the face tracking mechanism with other functionalities. Currently, I haven't cared much about the integration with other modules. For example, I need to auto-disable face tracking in the large video when it’s muted. I also plan to auto-resize face prompts when full screen mode is toggled.
Rewrite tracking.js library. My current work involves in adding new features in original tracking.js library, mainly in
builddirectory. While, this is not a good coding practice to directly inject new code there. The problem is that tracking.js library doesn't support CommonJS or ES6 modules, which makes it difficult to add new functionality. I plan to rewrite the library in ES6 modules, as well as adding new necessary features.