分布式视频转码系统的设计与完成（二）--Hadoop集群上ffmpeg的配置

首先去官网下载适合Linux系统的压缩包，不敢下载最新版本的文件，下载的是ffmpeg-3.3.8：http://www.ffmpeg.org/download.htm

 

在master上配置ffmpeg
在/usr文件下打开终端  sudo mkdir ffmpeg
在/usr文件下打开终端   sudo chown -R hadoop ./ffmpeg
将下载好的包粘贴到ffmpeg文件下，然后
tar -xjf ffmpeg-3.3.8.tar.bz2

sudo apt-get install yasm 
sudo apt-get install libx264-dev
1. sudo apt-get install libfaac-dev
2. sudo apt-get install libmp3lame-dev
3. sudo apt-get install libtheora-dev
4. sudo apt-get install libvorbis-dev
5. sudo apt-get install libxvidcore-dev
6. sudo apt-get install libxext-dev
7. sudo apt-get install libxfixes-dev

在解压好的文件夹ffmpeg-3.3.8中
./configure --enable-gpl --enable-version3 --enable-nonfree --enable-postproc  --enable-pthreads  --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-x11grab --enable-libvorbis --enable-shared --enable-yasm

 

   最后 编译安装 ffmpeg 

　　　　$ cd ffmpeg-3.3.8

　　　　$ sudo -s

　　　　$ source configure

　　　　$ make

　　　　$ make install

最后在/usr/local/bin 中运行ffmpeg -version命令报错

hadoop@master:/usr/local/bin$ ffmpeg version
ffmpeg: error while loading shared libraries: libavdevice.so.57: cannot open shared object file: No such file or directory
 

通过在ffmpeg-3.3.8文件夹下：

root@master:/usr/ffmpeg/ffmpeg-3.3.8# ldconfig

（这条命令是写入config文件配置）

 

最后效果：

hadoop@master:~$ ffmpeg -version
ffmpeg version 3.3.8 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04)
configuration: 
libavutil      55. 58.100 / 55. 58.100
libavcodec     57. 89.100 / 57. 89.100
libavformat    57. 71.100 / 57. 71.100
libavdevice    57.  6.100 / 57.  6.100
libavfilter     6. 82.100 /  6. 82.100
libswscale      4.  6.100 /  4.  6.100
libswresample   2.  7.100 /  2.  7.100
 

 

最后简单检查一下转码的效果：

hadoop@master:~$ ffmpeg -i /home/hadoop/桌面/test.mp4 -vcodec copy /home/hadoop/桌面/test.flv
ffmpeg version 3.3.8 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04)
  configuration: 
  WARNING: library configuration mismatch
  avutil      configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis --enable-shared --enable-yasm
  avcodec     configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis --enable-shared --enable-yasm
  avformat    configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis --enable-shared --enable-yasm
  avdevice    configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis --enable-shared --enable-yasm
  avfilter    configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis --enable-shared --enable-yasm
  swscale     configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis --enable-shared --enable-yasm
  swresample  configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis --enable-shared --enable-yasm
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/hadoop/桌面/test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.41.100
  Duration: 00:15:00.00, start: 0.000000, bitrate: 1094 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x360, 1040 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 48 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (aac (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, flv, to '/home/hadoop/桌面/test.flv':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.71.100
    Stream #0:0(und): Video: h264 (High) ([7][0][0][0] / 0x0007), yuv420p, 640x360, q=2-31, 1040 kb/s, 25 fps, 25 tbr, 1k tbn, 12800 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 44100 Hz, stereo, fltp (default)
    Metadata:
      handler_name    : SoundHandler
      encoder         : Lavc57.89.100 libmp3lame
frame=  688 fps=0.0 q=-1.0 size=    4854kB time=00:00:27.40 bitrate=1451.0kbits/frame= 1382 fps=1373 q=-1.0 size=    9173kB time=00:00:55.16 bitrate=1362.2kbitsframe= 2085 fps=1384 q=-1.0 size=   13422kB time=00:01:23.28 bitrate=1320.3kbitsframe= 2776 fps=1383 q=-1.0 size=   17313kB time=00:01:50.92 bitrate=1278.6kbitsframe= 3436 fps=1370 q=-1.0 size=   21873kB time=00:02:17.32 bitrate=1304.8kbitsframe= 4088 fps=1359 q=-1.0 size=   26768kB time=00:02:43.40 bitrate=1342.0kbitsframe= 4744 fps=1352 q=-1.0 size=   31772kB time=00:03:09.64 bitrate=1372.5kbitsframe= 5434 fps=1355 q=-1.0 size=   36211kB time=00:03:37.24 bitrate=1365.5kbitsframe= 6119 fps=1357 q=-1.0 size=   40915kB time=00:04:04.64 bitrate=1370.1kbitsframe=14434 fps=1311 q=-1.0 size=   88928kB time=00:09:37.24 bitrate=1262.0kbitsframe=15085 fps=1309 q=-1.0 size=   91525kB time=00:10:03.28 bitrate=1242.8kbitsframe=15672 fps=1303 q=-1.0 size=   94283kB time=00:10:26.76 bitrate=1232.3kbitsframe=16304 fps=1302 q=-1.0 size=   97656kB time=00:10:52.04 bitrate=1226.9kbitsframe=16933 fps=1300 q=-1.0 size=  100049kB time=00:11:17.20 bitrate=1210.3kbitsframe=17504 fps=1294 q=-1.0 size=  102958kB time=00:11:40.04 bitrate=1204.8kbitsframe=18110 fps=1291 q=-1.0 size=  105515kB time=00:12:04.28 bitrate=1193.4kbitsframe=18725 fps=1289 q=-1.0 size=  109291kB time=00:12:28.88 bitrate=1195.5kbitsframe=19310 fps=1285 q=-1.0 size=  113008kB time=00:12:52.28 bitrate=1198.7kbitsframe=19891 fps=1281 q=-1.0 size=  116892kB time=00:13:15.52 bitrate=1203.7kbitsframe=20522 fps=1280 q=-1.0 size=  119600kB time=00:13:40.76 bitrate=1193.7kbitsframe=21114 fps=1277 q=-1.0 size=  122790kB time=00:14:04.44 bitrate=1191.2kbitsframe=21707 fps=1275 q=-1.0 size=  126360kB time=00:14:28.16 bitrate=1192.3kbitsframe=22316 fps=1273 q=-1.0 size=  128853kB time=00:14:52.52 bitrate=1182.7kbitsframe=22500 fps=1272 q=-1.0 Lsize=  129332kB time=00:14:59.97 bitrate=1177.2kbits/s speed=50.9x    
video:114292kB audio:14062kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.762152%

通过播放视频，也会发现没有任何差异。

接着重复上述操作，在计算节点上均配置好我们的ffmpeg工具即可

 

出现的问题：上述操作命令在slave1上执行后失败，找不到ffmpeg命令

更换成这两个博客的方法

https://blog.csdn.net/leezha/article/details/77849286

https://blog.csdn.net/shitangdejiaozi/article/details/59110384

 

一、安装各种依赖包

1、yasm（libx264需要依赖yasm）

      sudo apt-get install yasm

2、libx264

      sudo apt-get install libx264-dev

3、libfaac

      sudo apt-get install libfaac-dev

4、libmp3lame

      sudo apt-get install libmp3lame-dev

5、libtheora

      sudo apt-get install libtheora-dev

6、libvorbis

      sudo apt-get install libvorbis-dev

7、libxvid

      sudo apt-get install libxvidcore-dev

8、libxext

      sudo apt-get install libxext-dev

9、libxfixes

      sudo apt-get install libxfixes-dev

 

二、下载FFmpeg安装包

官网下载最新版本的FFmpeg配置的时候不成功，最终使用的是FFmpeg 2.4.13 "Fresnel"，下载网址：http://www.ffmpeg.org/olddownload.html

1、解压

      tar -xjvf ffmpeg-2.4.13.tar.bz2

2、配置

      cd ffmpeg-2.4.13/

      ./configure --prefix=/usr/local/ffmpeg --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libfaac --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-x11grab --enable-libvorbis

3、编译安装

      make

      make install

 

接着

sudo ldconfig

 

最后

为了在任何地方都可以直接用ffmpeg运行，不用使用./ffmpeg 
，可以将可执行程序复制到bin目录下

sudo cp /usr/local/ffmpeg/bin/ffmpeg /usr/local/bin/ 
sudo cp /usr/local/ffmpeg/bin/ffprobe /usr/local/bin/ 
sudo cp /usr/local/ffmpeg/bin/ffserver /usr/local/bin/

检查功能效果： 

 hadoop@slave1:~$ ffmpeg -i /home/hadoop/桌面/test.mp4 -vcodec copy /home/hadoop/桌面/test.avi
ffmpeg version 3.3.8 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04)
  configuration: --prefix=/usr/local/ffmpeg --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --enable-libvorbis
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/hadoop/桌面/test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.41.100
  Duration: 00:15:00.00, start: 0.000000, bitrate: 1094 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x360, 1040 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 48 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (aac (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, avi, to '/home/hadoop/桌面/test.avi':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    ISFT            : Lavf57.71.100
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x360, q=2-31, 1040 kb/s, 25 fps, 25 tbr, 50 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: mp3 (libmp3lame) (U[0][0][0] / 0x0055), 44100 Hz, stereo, fltp (default)
    Metadata:
      handler_name    : SoundHandler
      encoder         : Lavc57.89.100 libmp3lame
frame=  685 fps=0.0 q=-1.0 size=    4835kB time=00:00:27.30 bitrate=1450.9kbframe= 1388 fps=1388 q=-1.0 size=    9182kB time=00:00:55.42 bitrate=1357.3kframe= 2080 fps=1386 q=-1.0 size=   13383kB time=00:01:23.10 bitrate=1319.3kframe= 2770 fps=1385 q=-1.0 size=   17239kB time=00:01:50.70 bitrate=1275.7kframe= 3467 fps=1386 q=-1.0 size=   22031kB time=00:02:18.58 bitrate=1302.3kframe= 4155 fps=1385 q=-1.0 size=   27269kB time=00:02:46.10 bitrate=1344.9kframe= 4849 fps=1385 q=-1.0 size=   32480kB time=00:03:13.86 bitrate=1372.5kframe= 5542 fps=1385 q=-1.0 size=   36835kB time=00:03:41.58 bitrate=1361.8kframe= 6221 fps=1382 q=-1.0 size=   41487kB time=00:04:08.74 bitrate=1366.3kframe= 6895 fps=1379 q=-1.0 size=   46258kB time=00:04:35.70 bitrate=1374.5kframe= 7561 fps=1374 q=-1.0 size=   50178kB time=00:05:02.34 bitrate=1359.6kframe= 8225 fps=1370 q=-1.0 size=   54370kB time=00:05:28.90 bitrate=1354.2kframe= 8901 fps=1369 q=-1.0 size=   59539kB time=00:05:55.94 bitrate=1370.3kframe= 9558 fps=1365 q=-1.0 size=   64357kB time=00:06:22.22 bitrate=1379.3kframe=10222 fps=1362 q=-1.0 size=   67237kB time=00:06:48.78 bitrate=1347.4kframe=10895 fps=1361 q=-1.0 size=   69208kB time=00:07:15.70 bitrate=1301.2kframe=11521 fps=1355 q=-1.0 size=   73674kB time=00:07:40.74 bitrate=1309.9kframe=12184 fps=1353 q=-1.0 size=   76087kB time=00:08:07.26 bitrate=1279.2kframe=12838 fps=1351 q=-1.0 size=   80265kB time=00:08:33.42 bitrate=1280.7kframe=13494 fps=1349 q=-1.0 size=   83992kB time=00:08:59.66 bitrate=1275.0kframe=14138 fps=1346 q=-1.0 size=   87693kB time=00:09:25.42 bitrate=1270.5kframe=14801 fps=1345 q=-1.0 size=   90148kB time=00:09:51.94 bitrate=1247.6kframe=15446 fps=1342 q=-1.0 size=   92743kB time=00:10:17.74 bitrate=1229.9kframe=16104 fps=1341 q=-1.0 size=   96608kB time=00:10:44.06 bitrate=1228.8kframe=16615 fps=1329 q=-1.0 size=   98540kB time=00:11:04.50 bitrate=1214.8kframe=17263 fps=1327 q=-1.0 size=  101734kB time=00:11:30.42 bitrate=1207.1kframe=17934 fps=1328 q=-1.0 size=  104384kB time=00:11:57.26 bitrate=1192.2kframe=18591 fps=1327 q=-1.0 size=  108146kB time=00:12:23.54 bitrate=1191.5kframe=19238 fps=1326 q=-1.0 size=  112229kB time=00:12:49.42 bitrate=1194.9kframe=19842 fps=1322 q=-1.0 size=  116297kB time=00:13:13.58 bitrate=1200.5kframe=20496 fps=1322 q=-1.0 size=  119177kB time=00:13:39.74 bitrate=1191.0kframe=21156 fps=1322 q=-1.0 size=  122748kB time=00:14:06.14 bitrate=1188.4kframe=21821 fps=1322 q=-1.0 size=  126700kB time=00:14:32.74 bitrate=1189.3kframe=22475 fps=1321 q=-1.0 size=  128937kB time=00:14:58.90 bitrate=1175.0kframe=22500 fps=1321 q=-1.0 Lsize=  130238kB time=00:14:59.97 bitrate=1185.5kbits/s speed=52.8x    
video:114292kB audio:14062kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.468033%

 

转码功能实现后：

我们做了一些转码测试：

2018年8月4日14:43:41
此时使用TranscoderMR项目导出的jar包为tc5.jar，已经可以完成分布式视频转码功能
1.但是在我们的MapReduce程序中设置了一些休眠的地方，是为了避免上一步的工作没有完成就直接执行下一步
Thread.sleep(10000);
2.我们将中间的一些转码参数注释掉了，只保留了对封装格式的修改

2018年8月4日15:15:06
导出的TranscoderMr2.jar中已经注释掉了Thread.sleep()，MapReduce程序能够正常运行，可以得到转码后的分片
但是注释掉的参数那一部分不能够恢复，这让人非常的心烦
下面这条指令是源代码中的指令（不能够成功执行）
ffmpeg -i /opt/DVTS/MRtest/split/Transcoding_0.mp4.split0001.mp4 -b:v 15000k -maxrate 30000k -bufsize 30000k -s 3840*2160 -r 25 -g 50 -pix_fmt yuv420p -aspect 16:9 -c:v libx265 -x265-params bframe=3:keyint=50 -c:a ac3 -ar 48k -b:a 256k -y /opt/DVTS/MRtest/trans/Transcoding_0.mp4.split0001.mp4.flv
下面这个命令能够成功执行
ffmpeg -i /opt/DVTS/MRtest/split/Transcoding_0.mp4.split0001.mp4 -b:v 15000k -maxrate 30000k -bufsize 30000k -s 3840*2160 -r 25 -g 50 -pix_fmt yuv420p -aspect 16:9 -c:v libx264 -x264-params bframes=3 -c:a ac3 -ar 48k -b:a 256k -y /opt/DVTS/MRtest/trans/Transcoding_0.mp4.split0001.mp4.ts
貌似我们的ffmpeg无法加载“libx265 -x265-params bframe=3:keyint=50”这段指令
我尝试了sudo apt-get install libx265-dev
但是不成功！

（这个问题应该可以被解决 ，需要再baidu，重新配置ffmpeg动态库就好了）可能就是之前ffmpeg的配置没有比较新的libx265库

上面这个命令还需要仔细解读一下才可以；

先将xml文件改成当前能调用转码命令的形式，在桌面，叫做Parma1.

这个也能成功转码，设置平均码率提高  分辨率不变  编码方式不变 封装格式改变
ffmpeg -i /opt/DVTS/MRtest/split/Transcoding_0.mp4.split0001.mp4 -b:v 1500k -maxrate 2000k -bufsize 2000k -s 640*360 -r 25 -g 50 -pix_fmt yuv420p -aspect 16:9 -c:v libx264 -x264-params bframes=3 -c:a ac3 -ar 48k -b:a 256k -y /opt/DVTS/MRtest/trans/Transcoding_0.mp4.split0001.mp4.avi
这个也能成功转码，设置平均码率提高  分辨率提高  编码方式改变 （像素点变多，模糊）
ffmpeg -i /opt/DVTS/MRtest/split/Transcoding_0.mp4.split0001.mp4 -b:v 1500k -maxrate 2000k -bufsize 2000k -s 1280*720 -r 25 -g 50 -pix_fmt yuv420p -aspect 16:9 -c:v mpeg4 -c:a ac3 -ar 48k -b:a 256k -y /opt/DVTS/MRtest/trans/Transcoding_0.mp4.split0001.mp4.avi
这个也能成功转码，设置平均码率提高  分辨率提高  编码方式改变 （像素点变少，也模糊）
ffmpeg -i /opt/DVTS/MRtest/split/Transcoding_0.mp4.split0001.mp4 -b:v 1500k -maxrate 2000k -bufsize 2000k -s 320*180 -r 25 -g 50 -pix_fmt yuv420p -aspect 16:9 -c:v mpeg4 -c:a ac3 -ar 48k -b:a 256k -y /opt/DVTS/MRtest/trans/Transcoding_0.mp4.split0001.mp4.avi