Augmented Reality Using C# and OpenCV

As I’ve mentioned before, my articles about using OpenCV and C# are the most viewed articles on this site. Among those articles, I get more emails asking about using OpenCV and C# for augmented reality applications than I do anything else. It appears that AR is a pretty big topic these days and everyone looking at getting into the field needs a good place to start. So, I’ve put together a small application that uses OpenCV and C# to do augmented reality. As always, I’m using the OpenCvSharp .NET wrapper for OpenCV. But, the same principles that apply here can also be used in pretty much any other wrapper or in OpenCV itself. Usually, I’ll walk thru every line of code in my example applications and explain what I’m doing. But, this time, I’ve decided to just provide you with the code and let you figure it out for yourself. However, as always, I’m more than willing to answer any questions you may have as you go along with the example.

To get started, you will need to download my OpenCV Augmented Reality example application fromhttp://www.prodigyproductionsllc.com/downloads/OpenCvAugmentedReality.zip. It already has everything you need to begin your augmented reality application including the OpenCvSharp and OpenCV runtimes which are located in the bin > Debug directory. In that same directory, you will see a file called “chessboard 6×5.jpg”. In order for this example to work, you will need to print a copy of that image onto a typical 8.5×11 piece of paper (scale doesn’t really matter here). Once you’ve printed the chessboard image, go ahead and launch the app by opening the .sln file in Visual C# or by running the AugmentedReality.exe file also found in the bin > Debug folder.

When you run the application, you will see a Windows form that only includes a button with the word “Start” on it and a combo box next to it. The combo box includes options for 1, 2, & 3. Picking number 1 means that the application will look for the checkerboard image in a video feed and will overlay a typical image over it. The image used in this example is a standard JPG file of the OpenCV logo. If you choose number 2 from the combo box, the application will look for the checkerboard in a video feed and will overlay a video over it. For this example, I’ve included a video file called “trailer.avi” which is a trailer for the “Big Buck Bunny” movie created by the Peach Open Movie Project (http://www.bigbuckbunny.org/). If you choose number 3 from the combo box, the application will look for the checkerboard image in a video feed and will draw a box around the portion of the image that is used to overlay the image or video. Here is an example of the application using number 2 to display the “Big Buck Bunny” trailer video over the checkerboard.

I’m using a cheapo USB webcam to capture the video and even running on low end laptop, I’m still getting pretty good performance out of the application. The possibilities of technology like this are endless. For example, since OpenCV can be ran on an iPhone, one could easily write an augmented reality iPhone app that can overlay advertisements on top of images when the iPhone is pointed at things like store fronts or billboards. Imagine pointing your phone at a still image in a magazine and having a video commercial play in your screen. You could do that with this kind of technology. Anyways, whatever you decide to use this for, be sure to come back here and share your story with the rest of us in the comments below. I’m extremely curious as to what all kinds of cool stuff you guys can come up with. Until next time, HAPPY CODING!!!