Tutorial:background detection with opencv

UPDATE: This tutorial is written for OpenCV 2.3 and as one of the commenters pointed out, from OpenCV 2.4 the nmixtures are protected instead of public, so you might have to set the values for nmixtures and bShadowDetection using the constructor.

This is a first post from what hopefully will become series of tutorials so stay tuned. Those tutorials aren't exactly intended for complete beginners, although I try to explain as much things as possible. I presume that you have at least basic grasp of C++ and have already configured OpenCV. Some knowledge about image processing also would help.

Lately, as a part of bigger project, I needed to detect a background in quite precise way. I didn't have time to write my own background estimation algorithm or reimplement existent, so I decided to use functions from OpenCV - the results are very good with very little effort (the code is about 35 lines long). The program itself does background, foreground segmentation and draws nice contours around detected foreground objects.

This is the listing of the whole code:

#include<opencv2/opencv.hpp>
#include<iostream>
#include<vector>

int main(int argc, char *argv[])
{
    cv::Mat frame;
    cv::Mat back;
    cv::Mat fore;
    cv::VideoCapture cap(0);
    cv::BackgroundSubtractorMOG2 bg;
    bg.nmixtures = 3;
    bg.bShadowDetection = false;

    std::vector<std::vector<cv::Point> > contours;

    cv::namedWindow("Frame");
    cv::namedWindow("Background");

    for(;;)
    {
        cap >> frame;
        bg.operator ()(frame,fore);
        bg.getBackgroundImage(back);
        cv::erode(fore,fore,cv::Mat());
        cv::dilate(fore,fore,cv::Mat());
        cv::findContours(fore,contours,CV_RETR_EXTERNAL,CV_CHAIN_APPROX_NONE);
        cv::drawContours(frame,contours,-1,cv::Scalar(0,0,255),2);
        cv::imshow("Frame",frame);
        cv::imshow("Background",back);
        if(cv::waitKey(30) >= 0) break;
    }
    return 0;
}

As you can see there's nothing too scary. Let's analyze it step by step.

#include<vector>
#include<opencv2/opencv.hpp>

We use vector as a container for our contours, so we need to include the vector library. The opencv.hpp is a header that contains all headers from OpenCV library - it's very convenient to include it during development but very unnecessary in final project, so keep in mind to change when app is ready.

Moving to the main function, firstly we have to create matrices which will contain our images and a capture handler.

cv::Mat frame; //(almost) raw frame
cv::Mat back; // background image
cv::Mat fore; // foreground mask
cv::VideoCapture cap(0); // capture video from camera

Next we need to declare our background detector - in this example I used gaussian mixture based detector. If you'd like to know more, here's the article describing the method. The initialization is pretty straightforward - everything has default values which needs only some fine-tuning. Usually when using a webcam there are no shadows visible, so you can turn shadow detection off.

cv::BackgroundSubtractorMOG2 bg; 
bg.nmixtures = 3; // set number of gaussian mixtures
bg.bShadowDetection = false; // turn the shadow detection off

Next thing is rather interesting. We declare a vector of vectors of points. Think about it for a minute. The detected contour is described as points, so we need a vector of points, but then we can have many contours so we need a vector of contours. If we connect it together we get vector of vectors of points - ta da!

std::vector<std::vector<cv::Point> > contours;

Because we are working with video streams, we need a loop to process all those frames. First thing is to grab a frame from handler and move it to a matrix called "frame":

cap >> frame;

Finally it is time to detect those foreground objects. The operator() function actually handles the updating of the background model and creates a binary mask of foreground objects. Additionally we write the background model image to the "back" matrix.

bg.operator ()(frame,fore);
bg.getBackgroundImage(back);

The mask can be quite noisy, so in order to get rid of the noise we do an opening. Opening is a morphological operation composed of two other morphological transformation: erosion and dilation (in that order).

cv::erode(fore,fore,cv::Mat());
cv::dilate(fore,fore,cv::Mat());

After detection it would be nice to detect and draw some contours. We use two functions to do this:

cv::findContours( fore, // binary input image 
                               contours, // vector of vectors of points
                               CV_RETR_EXTERNAL, // retrieve only external contours
                               CV_CHAIN_APPROX_NONE); // detect all pixels of each contour

cv::drawContours( frame, // draw contours here
                                  contours, // draw these contours
                                  -1, // draw all contours
                                  cv::Scalar(0,0,255), // set color
                                  2); // set thickness

To see the effects of our diligent work, we need to push the frames to the created windows. Also, we need to provide some way to end the program - we do it by using cv::waitKey() function. The function waits given number of miliseconds for a key hit and records it. Using simple if we wait for a key and break the loop afterwards.

cv::imshow("Frame",frame);
cv::imshow("Background",back);
if(cv::waitKey(30) >= 0) break;

And that is all what we need. Now it is time for some action shots.