With many of us working from home due to recent COVID-19 pandemic, there is increase in usage of video conferencing apps like Microsoft Teams and Zoom

Edit live video background with WebRTC and TensorFlow.js

submited by

Style Pass

2021-05-25 03:00:06

With many of us working from home due to recent COVID-19 pandemic, there is increase in usage of video conferencing apps like Microsoft Teams and Zoom.

One of the features that they provide is custom background/background blur to the video stream. Zoom being the popular video conferencing app offers this feature via chroma key i.e. For a better quality it requires a green screen or a clear background to distinguish the user with the background.

We can blur the background in the stream by applying deep learning without any need for green screen so that we need not spend time on clearing the workspace before connecting to a call 😉

v4l2loopback can be used to create virtual video source. Frames from the video stream can be read with OpenCV. Once we have the video frame as an image, the next step is to detect the background from the image for which we can apply semantic segmentation.

Semantic segmentation segments an image by associating each pixel of an object with the class label so that we can get exact pixels for the background and the person for our use case.