opencv+ffmpeg+QOpenGLWidget开发的音视频播放器demo

前言

    本篇文档的demo包含了 1.使用OpenCV对图像进行处理，对图像进行置灰，旋转，抠图，高斯模糊，中值滤波，部分区域清除置黑，背景移除，边缘检测等操作；2.单纯使用opencv播放显示视频；3.使用opencv和openGL播放显示视频；4.在ffmpeg解码后，使用opencv显示视频，并支持对视频的旋转翻转、裁剪、添加文字、添加logo、亮度调节、置灰、录像截图，音频开关等功能。视频播放器同时支持本地文件与网络码流地址的播放。本篇博客的最后有提供工程代码的下载。

一、OpenCV简单介绍：

    OpenCV是一个基于Apache2.0许可（开源）发行的跨平台计算机视觉和机器学习软件库，可以运行在Linux、Windows、Android和Mac OS操作系统上。它由一系列 C 函数和少量 C++ 类构成，同时提供了Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。应用领域包含：人机互动、物体识别、图像分割、人脸识别、动作识别、运动跟踪、机器人、运动分析、机器视觉、结构分析、汽车安全驾驶等。

二、demo调用OpenCV：

    本篇demo使用的是OpenCV-4.5.1, 是直接下载windows版本，没有额外去编译OpenCV的库，安装exe后，提取头文件和相关的库文件，用于工程调用即可。OpenCV下载路径：https://sourceforge.net/projects/opencvlibrary/

    OpenCV常用函数介绍：
    1）Mat Image = imread(filename); //从指定文件加载图像并返回
    2）namedWindow(“IMG-WIN”, cv::WINDOW_NORMAL);//创建图像显示窗口
    3）destroyWindow(“IMG-WIN”);//销毁图像显示窗口
    4）imshow(“IMG-WIN”, Image);//将图片显示在窗口中
    5）imwrite将图像保存到指定文件
    6）其他

三、demo功能介绍：

3.1.图像操作

    图像操作包含了显示原图、置灰、旋转、抠图、高斯模糊、中值滤波，部分区域清除置黑，背景移除，边缘检测等操作。点击显示原图后，通过点击其他按钮，可以实现图像的不同效果，例如置灰：

    其他功能可下载demo进行操作，点击显示原图，可恢复到之前图片显示，点击保存图片按钮可将处理后的图片进行保存。

3.2.图像操作相关代码

//图像操作
/
void MainWindow::on_pushButton_showImage_clicked()
{
    // load image using opencv
    m_srcImage = imread("./Image/test1.jpg");

    std::string winName = "Display window";
    namedWindow(winName, WINDOW_AUTOSIZE);
    imshow(winName, m_srcImage);

    if(m_bShowImage == false)
    {
        HWND hwnd = static_cast(cvGetWindowHandle(winName.c_str()));
        HWND parent = GetParent(hwnd);//得到nameWindow窗口的父句柄
        SetParent(hwnd, (HWND)ui->widget_image->winId());//设置ui控件的句柄是父句柄
        ShowWindow(parent, SW_HIDE);//隐藏掉nameWindow窗口

        resizeWindow(winName, cv::Size(ui->widget_image->width(), ui->widget_image->height()));
    }

    m_destImage = m_srcImage;
    m_bShowImage = true;
}

void MainWindow::on_pushButton_saveImage_clicked()
{
    if(m_bShowImage)
    {
        std::string saveFile = "./Image/test-new.jpg";
        if(imwrite(saveFile, m_destImage))
        {
            QMessageBox::information(this, "myOpencv", "Save Image Success.");
        }
        else
        {
            QMessageBox::critical(this, "myOpencv", "Save Image Failed.");
        }
    }
    else
        QMessageBox::critical(this, "myOpencv", "Please click on the Show Original Image button first.");
}

void MainWindow::on_pushButton_gray_clicked()
{
    if(m_bShowImage)
    {
        cvtColor(m_srcImage, m_destImage, COLOR_BGR2GRAY);

        std::string winName = "Display window";
        namedWindow(winName, WINDOW_AUTOSIZE);
        imshow(winName, m_destImage);
    }
    else
        QMessageBox::critical(this, "myOpencv", "Please click on the Show Original Image button first.");
}

void MainWindow::on_pushButton_rotate_clicked()
{
    if(m_bShowImage)
    {
        warpAffine(m_srcImage, m_destImage, getRotationMatrix2D({m_srcImage.cols/2.0f, m_srcImage.rows/2.0f},
                                                            45,0.5),m_srcImage.size(),cv::INTER_LINEAR, cv::BORDER_CONSTANT, {255,0,0});
        std::string winName = "Display window";
        namedWindow(winName, WINDOW_AUTOSIZE);
        imshow(winName, m_destImage);
    }
    else
        QMessageBox::critical(this, "myOpencv", "Please click on the Show Original Image button first.");
}

void MainWindow::on_pushButton_matting_clicked()
{
    if(m_bShowImage)
    {
        m_destImage = m_srcImage({20, 20, 60, 60}).clone();

        std::string winName = "Display window";
        namedWindow(winName, WINDOW_AUTOSIZE);
        imshow(winName, m_destImage);
    }
    else
        QMessageBox::critical(this, "myOpencv", "Please click on the Show Original Image button first.");
}

void MainWindow::on_pushButton_gauss_clicked()
{
    if(m_bShowImage)
    {
        GaussianBlur(m_srcImage, m_destImage, cv::Size(5,5), 0);

        std::string winName = "Display window";
        namedWindow(winName, WINDOW_AUTOSIZE);
        imshow(winName, m_destImage);
    }
    else
        QMessageBox::critical(this, "myOpencv", "Please click on the Show Original Image button first.");
}

void MainWindow::on_pushButton_median_clicked()
{
    if(m_bShowImage)
    {
        medianBlur(m_srcImage, m_destImage, 5);

        std::string winName = "Display window";
        namedWindow(winName, WINDOW_AUTOSIZE);
        imshow(winName, m_destImage);
    }
    else
        QMessageBox::critical(this, "myOpencv", "Please click on the Show Original Image button first.");
}

void MainWindow::on_pushButton_clear_clicked()
{
    if(m_bShowImage)
    {
        cv::Rect roi(40, 40, 80, 80);
        Mat roiMat = m_destImage(roi);
        roiMat.setTo(cv::Scalar(0,0,0));

        std::string winName = "Display window";
        namedWindow(winName, WINDOW_AUTOSIZE);
        imshow(winName, m_destImage);
    }
    else
        QMessageBox::critical(this, "myOpencv", "Please click on the Show Original Image button first.");
}

void MainWindow::on_pushButton_clearBackground_clicked()
{
    //原图
    Mat image = imread("./Image/test1.jpg");

    // 创建一个掩码（mask），用于指定哪些区域是前景，哪些区域是背景
    cv::Mat mask(image.size(), CV_8UC1, cv::Scalar(cv::GC_BGD));
    cv::Rect rectangle(50, 50, image.cols - 100, image.rows - 100);
    cv::grabCut(image, mask, rectangle, cv::Mat(), cv::Mat(), 5, cv::GC_INIT_WITH_RECT);

    // 将掩码中被标记为前景的像素设为白色，背景设为黑色
    cv::Mat foregroundMask = (mask == cv::GC_PR_FGD) | (mask == cv::GC_FGD);
    cv::Mat foregroundImage(image.size(), CV_8UC3, cv::Scalar(0, 0, 0));
    image.copyTo(foregroundImage, foregroundMask);

    // 显示结果图像
    cv::imshow("Foreground", foregroundImage);
    cv::waitKey(0);
}

void MainWindow::on_pushButton_edgeDetection_clicked()
{
    // 读取图片
    cv::Mat image = cv::imread("./Image/test1.jpg", cv::IMREAD_GRAYSCALE);

    // 边缘检测
    cv::Mat edges;
    cv::Canny(image, edges, 100, 200);

    // 显示结果
    cv::imshow("Edges", edges);
    cv::waitKey(0);
}

3.3.视频操作

    代码中通过宏定义可以旋转不同的视频处理方式（默认使用ffmpeg解码后，opencv播放显示视频），如下：

#define OPENCV_PLAY_VIDEO                   0   //单纯使用opencv播放显示视频
#define OPENCV_OPENGL_PLAY_VIDEO            0   //使用opencv和openGL播放显示视频
#define OPENCV_FFMPEG_PLAY_VIDEO            1   //使用ffmpeg解码后，opencv显示视频

    播放器支持对视频的旋转翻转、裁剪、添加文字、添加logo、亮度调节、置灰、录像截图，音频开关等功能，同时支持本地文件与网络码流地址的播放。

    简单演示如下：

    对原视频进行相关处理后，可通过点击录像截图按钮进行视频与图片的保存。通过开关音频按钮，支持对音频的播放处理。

    demo工程中有提供测试视频和测试logo，可以进行添加测试。

3.4.视频操作部分代码

void MainWindow::on_pushButton_Open_clicked()
{
    QString sFileName = QFileDialog::getOpenFileName(this, QString::fromLocal8Bit("Select Video File"));
    if(sFileName.isEmpty())
    {
        QMessageBox::critical(this, "myOpencv", QObject::tr("错误:文件不能为空."));
        return;
    }

    ui->lineEdit_url->setText(sFileName);
}

void MainWindow::on_pushButton_Play_clicked()
{
    QString sStreamAddr = ui->lineEdit_url->text();
    if(sStreamAddr.isEmpty())
    {
        QMessageBox::critical(this, "myOpencv", QObject::tr("错误:码流地址不能为空."));
        return;
    }

#if OPENCV_FFMPEG_PLAY_VIDEO
    if(nullptr == m_pMediaThread)
    {
        MY_DEBUG << "new MediaThread";
        m_pMediaThread = new MediaThread(ui->openGLWidget);
    }
    else
    {
        if(m_pMediaThread->isRunning())
        {
            QMessageBox::critical(this, "myOpencv", "错误:请先点击停止按钮关闭视频.");
            return;
        }
    }

    bool bMediaInit = m_pMediaThread->Init(sStreamAddr);
    if(bMediaInit)
    {
        m_pMediaThread->startThread();
    }

#elif OPENCV_PLAY_VIDEO
    if(m_pOpencvPlayThread1 == nullptr)
    {
        m_pOpencvPlayThread1 = new COpencvPlayThread1(ui->openGLWidget, sStreamAddr);
    }
    m_pOpencvPlayThread1->startThread();

#elif OPENCV_OPENGL_PLAY_VIDEO
    if(m_pOpencvPlayThread2 == nullptr)
    {
        m_pOpencvPlayThread2 = new COpencvPlayThread2(ui->openGLWidget, sStreamAddr);
    }
    m_pOpencvPlayThread2->startThread();

#endif

}

void MainWindow::on_pushButton_Stop_clicked()
{
#if OPENCV_PLAY_VIDEO
    if(m_pOpencvPlayThread1)
        m_pOpencvPlayThread1->stopThread();
#endif

#if OPENCV_OPENGL_PLAY_VIDEO
    if(m_pOpencvPlayThread2)
        m_pOpencvPlayThread2->stopThread();
#endif

#if OPENCV_FFMPEG_PLAY_VIDEO
    if(m_pMediaThread)
    {
        disconnect(m_pMediaThread, SIGNAL(sig_emitImage(const QImage&)),
                ui->openGLWidget, SLOT(slot_showImage(const QImage&)));

        m_pMediaThread->stopThread();
        m_pMediaThread->quit();
        m_pMediaThread->wait();
        m_pMediaThread->DeInit();
    }
#endif

}

void MainWindow::on_pushButton_startRecord_clicked()
{
    if(m_pMediaThread)
        m_pMediaThread->startRecord();
}

void MainWindow::on_pushButton_stopRecord_clicked()
{
    if(m_pMediaThread)
    {
        m_pMediaThread->stopRecord();
        QDesktopServices::openUrl(QUrl::fromLocalFile(RECORD_DEFAULT_PATH));
    }
}

void MainWindow::on_pushButton_snapshot_clicked()
{
    if(m_pMediaThread)
    {
        m_pMediaThread->Snapshot();
        QDesktopServices::openUrl(QUrl::fromLocalFile(SNAPSHOT_DEFAULT_PATH));
    }
}

void MainWindow::on_pushButton_video_rotate_clicked()
{
    if(m_pMediaThread)
    {
        int nRotate = ui->comboBox_rotate->currentIndex() - 1;
        if(nRotate < 0)
            m_pMediaThread->setRotate(false, nRotate);
        else
            m_pMediaThread->setRotate(true, nRotate);
    }
}

void MainWindow::on_pushButton_video_overturn_clicked()
{
    if(m_pMediaThread)
    {
        int nOverturn = ui->comboBox_overturn->currentIndex() - 1;
        if(nOverturn < 0)
            m_pMediaThread->setFlip(false, nOverturn);
        else
            m_pMediaThread->setFlip(true, nOverturn);
    }
}

bool bGrey = false;
void MainWindow::on_pushButton_video_grey_clicked()
{
    if(m_pMediaThread)
    {
        if(bGrey)
        {
            m_pMediaThread->setGray(false);
            bGrey = false;
        }
        else
        {
            m_pMediaThread->setGray(true);
            bGrey = true;
        }
    }
}

void MainWindow::on_pushButton_video_cropping_clicked()
{
    if(m_pMediaThread)
    {
        int x = ui->lineEdit_crop_X->text().toInt();
        int y = ui->lineEdit_crop_Y->text().toInt();
        int w = ui->lineEdit_crop_W->text().toInt();
        int h = ui->lineEdit_crop_H->text().toInt();

        m_pMediaThread->setCrop(true, x, y, w, h);
    }
}

void MainWindow::on_pushButton_video_addText_clicked()
{
    if(m_pMediaThread)
    {
        int x = ui->lineEdit_text_X->text().toInt();
        int y = ui->lineEdit_text_Y->text().toInt();
        int nColor = ui->comboBox_text_rgb->currentIndex();
        double size = ui->comboBox_text_fonsize->currentText().toDouble();
        QString sText = ui->lineEdit_videoText->text();

        MY_DEBUG << "size:" << size;
        m_pMediaThread->setAddText(true, sText, x, y, nColor, size);
    }
}

void MainWindow::on_pushButton_logoPath_clicked()
{
    QString sFileName = QFileDialog::getOpenFileName(this, QString::fromLocal8Bit("Select Logo File"));
    if(sFileName.isEmpty())
    {
        QMessageBox::critical(this, "myOpenCV", QObject::tr("错误:文件不能为空."));
        return;
    }

    ui->lineEdit_logo_path->setText(sFileName);
}

void MainWindow::on_pushButton_addLogo_clicked()
{
    if(m_pMediaThread)
    {
        int x = ui->lineEdit_logo_X->text().toInt();
        int y = ui->lineEdit_logo_Y->text().toInt();
        QString sAddLogoPath = ui->lineEdit_logo_path->text();

        m_pMediaThread->setAddLogo(true, x, y, sAddLogoPath);
    }
}

void MainWindow::on_pushButton_setBrightness_clicked()
{
    if(m_pMediaThread)
    {
        double brightness = ui->comboBox_brightnessVal->currentText().toDouble();
        m_pMediaThread->setImagePara(brightness);
    }
}

void MainWindow::on_pushButton_restore_clicked()
{
    if(m_pMediaThread)
    {
        int x = ui->lineEdit_crop_X->text().toInt();
        int y = ui->lineEdit_crop_Y->text().toInt();
        int w = ui->lineEdit_crop_W->text().toInt();
        int h = ui->lineEdit_crop_H->text().toInt();

        m_pMediaThread->setCrop(false, x, y, w, h);
    }
}

void MainWindow::on_pushButto_video_deleteText_clicked()
{
    if(m_pMediaThread)
    {
        int x = ui->lineEdit_text_X->text().toInt();
        int y = ui->lineEdit_text_Y->text().toInt();
        int nColor = ui->comboBox_text_rgb->currentIndex();
        double size = ui->comboBox_text_rgb->currentText().toDouble();
        QString sText = ui->lineEdit_videoText->text();

        m_pMediaThread->setAddText(false, sText, x, y, nColor, size);
    }
}

void MainWindow::on_pushButton_deleteLogo_clicked()
{
    if(m_pMediaThread)
    {
        int x = ui->lineEdit_logo_X->text().toInt();
        int y = ui->lineEdit_logo_Y->text().toInt();
        QString sAddLogoPath = ui->lineEdit_logo_path->text();

        m_pMediaThread->setAddLogo(false, x, y, sAddLogoPath);
    }
}

void MainWindow::on_pushButton_audioOpen_clicked()
{
    ctAudioPlayer::getInstance().isPlay(true);
}

void MainWindow::on_pushButton_audioClose_clicked()
{
    ctAudioPlayer::getInstance().isPlay(false);
}

void ctFFmpeg::dealWithOpenCV(AVFrame *pFrame/*, int time*/)
{
    if(!m_bQuit)
    {
        cv::Mat destCvMat(cv::Size(m_nVideoW, m_nVideoH), CV_8UC3, cv::Scalar(1, 2, 3));
        destCvMat.data = (uchar*)pFrame->data[0];

        //旋转
        if(m_bRotate)
        {
            MY_DEBUG << "Set Rotate m_nRotate:" << m_nRotate;
            rotate(destCvMat, destCvMat, m_nRotate);
        }

        //翻转
        if(m_bFlip)
        {
            MY_DEBUG << "Set Flip";
            flip(destCvMat, destCvMat, m_nFlip);
        }

        //裁剪
        if(m_bCrop)
        {
            destCvMat = destCvMat(Rect(m_nCropX, m_nCropY, m_nCropW, m_nCropH));
        }

        //灰度
        if(m_bGray)
        {
            MY_DEBUG << "Set Gray";
            cvtColor(destCvMat, destCvMat, COLOR_BGR2GRAY);
        }

        //添加文字
        if(m_bAddText)
        {
            MY_DEBUG << "m_nAddTextX:" << m_nAddTextX << ", m_nAddTextY：" << m_nAddTextY << ",m_nAddTextSize:" << m_nAddTextSize;
            cv::Point ptPos(m_nAddTextX, m_nAddTextY);
            int fontFace = cv::FONT_HERSHEY_COMPLEX;
            double fontScale = m_nAddTextSize;
            int thickness = 2;
            cv::Scalar color;

            switch(m_nAddTextColor)
            {
                case ADD_TEXT_COLOR_TYPE_BLUE:
                    color[0]=255; color[1]=0; color[2]=0;
                    cv::putText(destCvMat, m_sAddText.toStdString(), ptPos, fontFace, fontScale, color, thickness);
                break;
                case ADD_TEXT_COLOR_TYPE_GREEN:
                    color[0]=0; color[1]=255; color[2]=0;
                    cv::putText(destCvMat, m_sAddText.toStdString(), ptPos, fontFace, fontScale, color, thickness);
                break;
                case ADD_TEXT_COLOR_TYPE_RED:
                default:
                    color[0]=0; color[1]=0; color[2]=255;//红色
                    cv::putText(destCvMat, m_sAddText.toStdString(), ptPos, fontFace, fontScale, color, thickness);
                break;
            }
        }

        //添加logo
        if(m_bAddLogo)
        {
            cv::Mat logoImage = cv::imread(m_sAddLogoPath.toStdString());
            cv::Mat roi = destCvMat(cv::Rect(m_nAddLogoX, m_nAddLogoY, logoImage.cols, logoImage.rows));
            cv::addWeighted(roi, 0.5, logoImage, 0.5, 0, roi);
        }

        //亮度\暗度
        cv::addWeighted(destCvMat, m_nBrightness, cv::Scalar(0, 0, 0), 0, 0, destCvMat);

#if OPENCV_NAME_WINDOW_SHOW
        cv::imshow(m_sWinName.toStdString(), destCvMat);
#endif

        //效率太慢, 容易导致崩溃, 不适合转QImage再用openGL播放
        //...
        //emit sig_getImage(image);

        //截图
        if(m_bSnapshot)
        {
            cv::cvtColor(destCvMat, destCvMat, cv::COLOR_BGR2RGB);

            QImage image(destCvMat.data, destCvMat.cols, destCvMat.rows, static_cast(destCvMat.step), QImage::Format_RGB888);

            QPixmap pixPicture = QPixmap::fromImage(image);

            QString sPath = "./snapshot/";
            QDate date = QDate::currentDate();
            QTime time = QTime::currentTime();
            m_sSnapPath = QString("%1%2-%3-%4-%5%6%7.jpg").arg(sPath).arg(date.year()). \
                    arg(date.month()).arg(date.day()).arg(time.hour()).arg(time.minute()). \
                    arg(time.second());
            MY_DEBUG << "Snapshot... m_sSnapPath:" << m_sSnapPath;
            pixPicture.save(m_sSnapPath, "jpg");
            m_bSnapshot = false;
        }
    }
}

四、工程源码下载：

demo播放器下载：https://download.csdn.net/download/linyibin_123/88217460

demo播放器工程源码下载：https://download.csdn.net/download/linyibin_123/88217472