Display OpenCV Image from Jupyter Notebook

Introduction

while ways of displaying OpenCV image (in cv::Mat format) on the local development machine has been discussed in the article Add Plotting Support For OpenCV C++, it's also noticeably important to be able to display the image from the front-end of a Jupyter notebook, which opens the possibility for developers to do remote programming.

This article aims to share how to set up the Jupyter environment with cling support so that those who are interested in OpenCV are able to create and debug online C++-based OpenCV programs and get runtime feedbacks from the execution results.

Environments

1. Download and Install MiniConda.

Note: MiniConda is required due to the conflict with the [ZeroMQ] library which is already installed in the anaconda distribution.

2. Create a new Conda environment for cling

conda create -n cling

for list of created environments, run below command

conda env list

Below is the sample output which indicates we are currently working on the cling environment.

# conda environments:
#
                         /Users/nling/opt/anaconda3
                         /Users/nling/opt/anaconda3/envs/cling
                         /Users/nling/opt/anaconda3/envs/opencv-course
base                     /Users/nling/opt/miniconda3
cling                 *  /Users/nling/opt/miniconda3/envs/cling
kotlin                   /Users/nling/opt/miniconda3/envs/kotlin

3. Activate the newly created env and install xeus-cling and required packages therein.

conda activate cling
conda install xeus-cling  -c conda-forge
conda install matplotlib  -c conda-forge
conda install numpy       -c conda-forge
conda install jupyterlab  -c conda-forge

Run the below command to check installed Jupyter kernels for the active environment.

jupyter kernelspec list

Below output indicates kernels for cpp11, 14, and 17 are available now.

Available kernels:
  python3    /Users/nling/opt/miniconda3/envs/cling/share/jupyter/kernels/python3
  xcpp11     /Users/nling/opt/miniconda3/envs/cling/share/jupyter/kernels/xcpp11
  xcpp14     /Users/nling/opt/miniconda3/envs/cling/share/jupyter/kernels/xcpp14
  xcpp17     /Users/nling/opt/miniconda3/envs/cling/share/jupyter/kernels/xcpp17

5. Launch the Jupyter web front and start creating a sample notebook.

jupyter-lab

Create Jupyter Notebook

1. Click on the Plus Sign (+) on the left top of the local homepage to launch the notebook launcher.
   Here C++17 was chosen as the C++ kernel.
   

   Create C++ powered notebook

2. Choose the notebook with the target kernel to start editing a new notebook.

   
   
   Sample code and the cell-by-cell execution result

References

1. SylvainCorlay, etc. Xeus-cling.
2. Jupyter Doc. Displaying rich content.
3. Pascal Thomet. Suggestion : add doc about OpenCV images display?. 2018/12/5, 2021/06/22.
4. Micah P. Dombrowski and Andrea Amantini. C++/Xeus Cling Environment. 2019/12/18, 2021/06/22.
5. Unknown. Advices and Gotchas when using cling and jupyter. 2021/06/22.

Appendix

• Source code of the notebook

#pragma cling add_include_path("$OPENCV_DIR/include/opencv4")
#pragma cling add_library_path("$OPENCV_DIR/lib")

#pragma cling load("opencv_imgcodecs")

#include "./displayImages.h"

using namespace cv;

displayImage("boy.jpg");

Mat img = imread("face.png");
displayImage(img);

• Source code of displayImages.h

#include 
#include 
#include 
#include "xtl/xbase64.hpp"
#include "xeus/xjson.hpp"
#include 

namespace im
{
    struct FileImage
    {   
        FileImage(const std::string& filename)
        {
            std::ifstream fin(filename, std::ios::binary);   
            m_buffer << fin.rdbuf();
        }
        
        std::stringstream m_buffer;
    };

    struct MatImage
    {
        MatImage(const cv::Mat& m) : _mat(m) {}
        cv::Mat _mat;
    };

    inline auto show(const std::string& filename)
    {
        return im::FileImage(filename);
    }

    inline auto show(const cv::Mat& img)
    {
        return im::MatImage(img);
    }
    
    xeus::xjson mime_bundle_repr(const FileImage& img)
    {
        auto bundle = xeus::xjson::object();
        bundle["image/png"] = xtl::base64encode(img.m_buffer.str());
        return bundle;
    }

    xeus::xjson mime_bundle_repr(const MatImage& img)
    {
        std::vector buf;
        bool success = cv::imencode(".png", img._mat, buf);
        if (success)
        {
            std::string bufString(buf.begin(), buf.end());
            auto bundle = xeus::xjson::object();
            bundle["image/png"] = xtl::base64encode(bufString);
            //bundle["text/plain"] = xtl::base64encode(bufString);
            return bundle;
        }
        else
        {
            return {};
        }
    }
}

void displayImage(const cv::Mat& image){
    xcpp::display(im::show(image));
}

void displayImage(const std::string& fileName){
    xcpp::display(im::show(fileName));
}