Virtual Camera for OpenCV using V4L2Loopback


Overview

Did you ever wonder how to create a virtual camera from a custom video source? Well, may be not, but nevertheless this is a very interesting project to do! Imagine that you can fetch the video from a RTSP source (or from a file), process it with OpenCV (adding some interesting filters), and then provide the result to the whole system, just like a normal USB webcam. Any other software (such as a web browser, Skype, Meet, Teams, Zoom…) could use the processed video seamlessly.

This is a recipe of how we could achieve this very result. Here, we will source the video feed from a USB webcam, convert it to grayscale (to give an example), and then send it to a video device, making it available for other software. We will open the result with Google Chrome, Cheese and Gstreamer.


Ingredients

For this tutorial, I used (and you will need):

  • A GNU/Linux system. I used Ubuntu 20.04 LTS

  • Basic knowledge of programming in C++ and OpenCV, Makefile (and also Python, JavaScript, HTML and CSS, if you want to delve a little bit deeper), plus some familiarity with system administration.

  • All the libraries and tools used in the example:

    • v4l2loopback-dkms
    • v4l2loopback-utils
    • gstreamer1.0-tools
    • gstreamer1.0-plugins-good
    • libopencv-dev
    • build-essential
    • google-chrome | chromium
    • vlc | cheese
  • A real and working camera (either integ

  • rated or USB will do)

I’ve created a repository with all the code explained in this recipe, so, if you want to download it using git:

git clone https://bitbucket.org/OscarAcena/ocv-virtual-cam

Warming Up

First of all, let me explain in more detail what we will do here. The source of the video is a normal webcam connected through USB, which means that the Video For Linux (v4l2) subsystem will take the control there. Then, using OpenCV, we will fetch the image data, frame by frame, and convert it to grayscale. Once we got the desired effect, we will write the frame to a virtual device that will be created by the module Video For Linux Loopback (v4l2loopback). Finally, we could open that output device using virtually any normal application that uses a video device (like Cheese, Gstreamer, VLC or even the web browser).

Let’s begin!


Setting Up: v4l2 loopback

The key component that made all of this possible is v4l2looback. This module creates virtual video devices that normal applications will read as if they were ordinary devices, but the video stream will not come from some piece of hardware, but instead from another application or service.

So, the first thing we have to do is create the virtual devices we’re going to use. As v4l2loopback is a kernel module, we do this by registering it inside the kernel with the proper params. Don’t worry, it’s easier than it looks. The parameters that we want to specify are:

  • the number of desired devices; here we will create two, one for an initial test using Gstreamer, and other one for the example using OpenCV.

  • the names and IDs of that devices; we will use IDs 5 and 6 (so /dev/video5 and /dev/video6 will be created), and for the names: “Gst VideoTest” and “OpenCV Camera“.

  • an option to use exclusive capabilities

This last parameter is worth an explanation. When you create a device with v4l2loopback, it could be read and written at the same time by any application, thus is a CAPTURE and an OUTPUT device at the same time. These capabilities are provided by the device as metadata, and the application will read it to check whether the device suits its needs. But, some applications are a bit picky, and wants only exclusive capabilities (or CAPTURE alone, or OUTPUT alone, but not both at the same time). This is the case with Google Chrome and WebRTC. Setting this flag will make the device announce as OUTPUT only (so it will not be seen by ordinary applications). When a producer is attached to the device, then it will change its capabilities to CAPTURE only, and then will be seen by the picky applications.

So, the command to do all of this is the following:

sudo modprobe v4l2loopback \
	devices=2 exclusive_caps=1,1 video_nr=5,6 \
	card_label="Gst VideoTest","OpenCV Camera"

Note that, as we create two devices, we need to specify the flag for each device independently. Otherwise, it will take the default value. You can check that everything worked with the command:

v4l2-ctl --list-devices -d5

Which should output something like this:

Gst VideoTest (platform:v4l2loopback-000):
	/dev/video5

OpenCV Camera (platform:v4l2loopback-001):
	/dev/video6

UVC Camera (046d:0825) (usb-0000:00:14.0-6.3):
	/dev/video0
	/dev/video1

There is a Makefile in the repository with rules to simplify some tasks in this recipe. So you can also run the following to achieve the same result:

make setup-devices

Initial Test: Gstreamer Producer

Strictly speaking, this step is not necessary, but it can help you try out if the previously created devices are working. We will launch a Video Test source and attach it to the first of our devices: /dev/video5. The easiest way to do this is using gstreamer, so just run the following command:

gst-launch-1.0 videotestsrc ! v4l2sink device=/dev/video5

You can also use the Makefile with the rule: start-gst-camera. In either case, it should display something like this:

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

Ok, the source is correctly bind, so now you can open that very same device with any application that can handle a webcam (or a video device). For instance, open Cheese or VLC. After choosing the right device (in case you have more than one), you will see something like this:

If you see it, congratulations! You have all ready for the next step!


Simple Filter with OpenCV

Now, what we will do is open a real webcam, acquire images from it, convert them to gray scale, and then write them to the other available device: /dev/video6. We will write a little program in C++ to do this task, so open your favorite editor and create a file for the program (for example, ocv-camera.cpp) and another one for a Makefile that will help you in the compilation process. If you feel yourself a bit lazy, these two files are also in the repository, ready to be used 🙂

In our program we will do the following actions:

  • Open the camera device for reading and configure it
  • Open the output device for writing and also configure it
  • Event loop to
    • Grab frame from camera
    • Convert frame to gray scale
    • Write frame to output device

Of course, you will need a main() function and some headers as well, so use the following basic skeleton to start with:

#include <opencv2/opencv.hpp>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/videodev2.h>

#define VID_WIDTH  640
#define VID_HEIGHT 480
#define VIDEO_IN   "/dev/video0"
#define VIDEO_OUT  "/dev/video6"

int
main(void) {
    // Here goes the real meat
}

Here, I’ve defined my Camera device as /dev/video0, and a basic resolution of 640×480. Feel free to change these constants to suit your needs.

Open and Configure Camera Device

Nothing too much special here, because, using OpenCV, this step is very easy. We’ll use a VideoCapture, changing the default width and height as specified by our constants:

// open and configure input camera (/dev/video0)
cv::VideoCapture cam(VIDEO_IN);
if (not cam.isOpened()) {
    std::cerr << "ERROR: could not open camera!\n";
    return -1;
}
cam.set(cv::CAP_PROP_FRAME_WIDTH, VID_WIDTH);
cam.set(cv::CAP_PROP_FRAME_HEIGHT, VID_HEIGHT);

Open and Configure Output Device

This step is a bit more elaborated. First, we open the device for read/write as we would do with any other file in the system:

// open output device
int output = open(VIDEO_OUT, O_RDWR);
if(output < 0) {
    std::cerr << "ERROR: could not open output device!\n" <<
    strerror(errno); return -2;
}

But this is not a common file, is a device file, and one handled by the V4L2 subsystem. So to configure it, we will need to call our good ol’ friend ioctl. So, we create a v4l2_format to hold all our configuration, and call ioctl a first time to fill the video format:

// acquire video format from device
struct v4l2_format vid_format;
memset(&vid_format, 0, sizeof(vid_format));
vid_format.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;

if (ioctl(output, VIDIOC_G_FMT, &vid_format) < 0) {
    std::cerr << "ERROR: unable to get video format!\n" <<
    strerror(errno); return -1;
}

Next, we can update some params according to our needs. Here we will set:

  • the correct width and height; as we will not change the size of our grabbed frame, it will match to it.
  • the used pixel format; there are many, I choose RGB.
  • the frame size; as I am using RGB (3 bytes per pixel), it will be width * height * 3.
  • the field type to none; no need for interlacing (for more info, see The Linux Kernel documentation, 3.7 Field Order)

One note about the pixel format: later, we will use WebRTC to open our virtual device, and not all pixel formats are supported (specially by Chrome). Choose one of YUV420, Y16, Z16, INVZ, YUYV, RGB24, MJPEG, JPEG. Thus, the code to setup the device will be:

// configure desired video format on device
size_t framesize = VID_WIDTH * VID_HEIGHT * 3;
vid_format.fmt.pix.width = cam.get(cv::CAP_PROP_FRAME_WIDTH);
vid_format.fmt.pix.height = cam.get(cv::CAP_PROP_FRAME_HEIGHT);
vid_format.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;
vid_format.fmt.pix.sizeimage = framesize;
vid_format.fmt.pix.field = V4L2_FIELD_NONE;

if (ioctl(output, VIDIOC_S_FMT, &vid_format) < 0) {
    std::cerr << "ERROR: unable to set video format!\n" <<
    strerror(errno); return -1;
}

Interlude: Simple GUI and Event Loop

This step is not necessary at all, but once again it may help you. I will display the acquired frame to check that everything until now is working, and for this I create an OpenCV window for later usage:

// create GUI window
const char* gui = "gui";
cv::namedWindow(gui);
cv::setWindowTitle(gui, "OpenCV test");

What you certainly need is an event loop, where all the following actions will take place over and over again. Something like this will do:

// loop over these actions:
while (true) {
    // loop actions
}

Event Loop: Grab Frame

We need a cv::Mat to hold the grabbed frame. Create it and grab the frame from our input device (alias cam):

// grab frame
cv::Mat frame;
if (not cam.grab()) {
    std::cerr << "ERROR: could not read from camera!\n";
    break;
}
cam.retrieve(frame);

Pretty straightforward. Yes, we could have just done cam >> frame;, but I’m from the good old school 😉

Event Loop: Convert Frame Gray Scale

It’s time to apply our super-special filter to the grabbed frame. Here, I’ve created another cv::Mat to hold the resulting image, and then apply a simple color conversion operation, from RGB to GRAY. Here arises a problem: GRAY is not a WebRTC supported format, so if I pretend to open it with Chrome, I’ll see nothing. Moreover, is not the format that I specified earlier, so it won’t work. Solution? Convert back again the color space to RGB (lost data will not be recovered again, so the resulting image will still be gray).

// apply simple filter
cv::Mat result;
cv::cvtColor(frame, result, cv::COLOR_RGB2GRAY);
cv::cvtColor(result, result, cv::COLOR_GRAY2RGB);

If you setup a GUI to see whats going on, here is a good place to show the frame: whichever you want, either the grabbed frame or the converted one:

// show frame
cv::imshow("gui", frame);

Event Loop: Write Frame Output Device

Lastly, with the proper frame ready, what remains is to write it to the output device. It is again a very common operation, using write():

size_t written = write(output, result.data, framesize);
if (written < 0) {
    std::cerr << "ERROR: could not write to output device!\n";
    close(output);
    break;
}

Now, your program is ready to be compiled and tested! To do so, open the Makefile you created before, and add the following lines:

CXXFLAGS   = `pkg-config --cflags opencv4`
LDLIBS     = `pkg-config --libs opencv4`

Just that. Then, make your application:

make ocv-camera
[...]
./ocv-camera

Upon launching it, you will see the captured frames from your camera (or the modified ones, depending on what you choose), and the device /dev/video6 will also be ready to use, as long as you maintain this application running. You can try it again with Cheese, VLC, Gstreamer or whatever application you want.


Opening with WebRTC (Chrome)

Along with the rest of the source code, I’ve created a simple web application that uses the mediaDevices interface provided by your navigator, to list and open all the available streams, including those created here (/dev/video5 and /dev/video6). Let’s review the code (inside the file sink.html).

HTML Component

The component that we will use to show the camera contents is the <video> element. As is, without any other properties; we will change it later using JavaScript. I’ve also added a <select> that will be populated with the list of devices:

<div id="content">
  <video></video>
  <select>
    <option value="">Select a camera</option>
  </select>
</div>

Enumerate Devices

Now, in JavaScript lands, we use the function mediaDevices.enumerateDevices() to list all available devices. Note that this is an asynchronous operation, so we must wait for it to finish before we could use the results (here, an await will do). The returned list contains all kind of devices (audio and video), so we need to filter it to get only the video type:

const devices = await navigator.mediaDevices.enumerateDevices();
const video_devs = devices.filter(device => device.kind == 'videoinput');

Once we got them, we can populate the HTML select created before, using the deviceId and label of each item:

let select = $('#content select');
video_devs.forEach(dev => {
   select.append(`<option value="${dev.deviceId}">${dev.label}</option>`);
});

Setting Up the Selected Device

When the user changes the input source in our <select>, we should update the source of the <video> element as well. So, we will use the provided deviceId, which is the value of the selected option:

let video = $('video')[0];
let option = select.find("option:selected");
let devid = option.attr('value');

With that deviceId, we will call mediaDevices.getUserMedia() (again, an asynchronous operation), which will return a Stream object that could be used as the source of the <video> element:

let constraints = {video: {deviceId: {exact: devid}}};
const stream = await navigator.mediaDevices.getUserMedia(constraints);
video.srcObject = stream;
video.play();

Of course, you should catch any exception that this method may arise, and do some other things to fully make the example work. You can peek the complete working example in the sink.html file.

Running

In order to check if everything works, we could just open the HTML file with the browser, but it won’t work. And this is because of security reasons: WebRTC only works well over HTTPS. So, to fix this problem, I’ve created a simple HTTP/SSL server with an auto-signed certificate. You can start it by issuing the following command (or make start-webserver):

./https-webserver.py

Now, open the suggested URL (https://localhost:4443/sink.html) with Google Chrome. It may ask you a couple of things:

  • first, it uses an invalid certificate, so you should accept the risk before opening the web.
  • second, to access your camera, it will request you permission.

To simplify this step, I’ve added a rule on Makefile that launches a Google Chrome in App mode, and ignores the certificate problem:

make open-web-sink

Please, note that the previous command will run Chrome in App Mode, and it may not ask you permissions to open the camera device. In that case, it will not work. Fortunately the fix is easy: open the URL with the browser as you normally do.

If you run the above command, you should see the following:

You can change the input source, and select the OpenCV Camera to admire your hard work!


References


Acknowledgements 

This work has been funded under the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 857159.

2 thoughts on “Virtual Camera for OpenCV using V4L2Loopback

  1. I thank you kindly for your efforts, it’s working for me.
    And while I did expect python, I was positively surprised that it isn’t that hard using
    opencv in cpp as well.
    I must admit though, that I didn’t get everything you did here in depth,
    mainly the v4l2 ioctl part…
    But yeah, all the nifty opencv stuff is ready to use and share to the world with our virtual cameras,
    so there is little to complain.
    Thanks!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s