在说这个m3u8下载库之前，首先得感谢github上面的qq494257084的m3u8Download项目，让我获取了灵感，同时也参考了这项目的部分代码。万分感谢前人的轮子。
吉特哈布地址：https://github.com/qq494257084/m3u8Download
回归正题，当时做这个下载器的目的就是为了抓取某些网站的电视剧以及电影下载观看，上期我们说到了我自己也封装了jsoup的框架，就是那框架让我抓到了不少m3u8是视频资源，一开始就打算用个WebView进行浏览就算了，但是后面想了一下，浏览器播放其实挺卡的，外出的时候也不方便观看。于是研究了一下m3u8的视频结构，就着手进行了这个m3u8视频下载器的开发。
首先，我们先来了解一下什么是m3u8视频文件。个人的理解就是m3u8文件相当于一个视频分片（ts格式的视频文件）列表索引，用来确定每个分片视频信息，视频分片的顺序以及加密信息等等。也可以说类似于Lrc格式的歌词文件。m3u8既可以用来作为直播，也可以用来作为点播，是一种不错的视频直播+点播的方案。直播的时候，服务器会不断更新m3u8文件，不断的生成视频分片（ts格式文件）；如果不是直播的时候，也可以当做一个视频分片的合集。
下载m3u8视频完整流程需要分为几步：
第一步：获取m3u8文件（相当于列表索引）；
第二步：解析m3u8文件，获取到视频分片是否有加密，加密方式是什么类型，分片的下载路径以及时长等等。
第三步：根据解析得到的信息，去下载路径把全部ts文件都下载下来（如果ts文件已加密，那就需要解密后写入文件）。
第四步：按照m3u8的文件索引顺序，将ts文件合并成一个mp4文件。
第五步：合并文件成功后，删除ts分片的视频文件。
前文提到的m3u8Download项目实际上已经解决了大部分的需求了，但是，我发现，该视频网站的m3u8文件里面同时包含有加密的ts文件和未加密的ts文件，这个m3u8Download项目框架就不够用了。所以我只能参考后，重新进行下载器的封装。
下图是整个框架的结构：
image.png
1.先把下载任务的监听器内容放出来。
public interface DownloadM3U8Listener {
/**
* 开始下载
*
* @param downloadModel
* @param noAd
*/
void startDownload(M3U8FileInfoModel downloadModel, boolean noAd);
/**
* 下载进度回调
*
* @param consume 耗时
* @param downloadUrl 下载地址
* @param finished 已完成的数量
* @param sum 总文件数量
* @param percent 下载百分比
* @param speedPerSecond 下载速度
* @param noAd 是否去广告
*/
void process(int consume, String downloadUrl, int finished, int sum, float percent, String speedPerSecond, boolean noAd);
/**
* 合并文件开始
*
* @param downloadModel
* @param noAd
*/
void mergeStart(M3U8FileInfoModel downloadModel, boolean noAd);
/**
* 合并文件结束
*
* @param downloadModel
* @param noAd
*/
void mergeEnd(M3U8FileInfoModel downloadModel, boolean noAd);
/**
* 删除ts文件开始
*
* @param downloadModel
* @param noAd
*/
void deleteStart(M3U8FileInfoModel downloadModel, boolean noAd);
/**
* 删除ts文件结束
*
* @param downloadModel
* @param noAd
*/
void deleteEnd(M3U8FileInfoModel downloadModel, boolean noAd);
/**
* 下载错误
*
* @param downloadModel
* @param e
* @param noAd
*/
void error(M3U8FileInfoModel downloadModel, Exception e, boolean noAd);
/**
* 下载完成（不管成功与否都会回调）
*
* @param downloadModel
* @param isSuccess
* @param totalSize 总文件数
* @param finishedCount 完成的数量
* @param noAd
*/
void complate(M3U8FileInfoModel downloadModel, boolean isSuccess, int totalSize, int finishedCount, boolean noAd);
}
2.定义下载的数据类，有单个m3u8文件的信息类（M3U8FileInfoModel）和单一部分的ts文件数据（M3U8TsPartInfoModel）
public class M3U8FileInfoModel {
//m3u8的最外层url
public String url;
//m3u8的真实内容URL
public String trueUrl;
//保存的文件名
public String fileName;
//ts文件部分的合集，为了防止一个m3u8文件出现多个加密算法
public ArrayList tsPartInfoModels = new ArrayList<>();
//合并后的文件存储目录
public String dir;
}
public class M3U8TsPartInfoModel {
//key的地址
public String keyUrl = "";
//加密方法
public String method = "";
//iv
public String iv = "";
//key
public String key = "";
//密钥字节
public byte[] keyBytes = new byte[16];
//key是否为字节
public boolean isByte = false;
//ts文件的集合
public Set tsSet = new LinkedHashSet<>();
public void addTsContent(String tsFileStr, double duration, int index) {
tsSet.add(new TsInfoModel(tsFileStr, duration, index));
}
public static class TsInfoModel {
/**
* 地址
*/
public String tsUrl = "";
/**
* 时长
*/
public double duration = 0;
/**
* 索引位置
*/
public int index = 0;
public TsInfoModel(String tsUrl, double duration, int index) {
this.tsUrl = tsUrl;
this.duration = duration;
this.index = index;
}
}
}
3.编写全局下载的管理类（M3U8DownloadManager），使用了线程池进行多任务下载管理。
public class M3U8DownloadManager {
private static M3U8DownloadManager m3u8Download;
//最大同时下载的M3U8文件，默认是3
private int maxDownloadTask = 3;
private ExecutorService fixedThreadPool;
private Set downloadTask = new HashSet<>();
public static M3U8DownloadManager getInstance() {
if (m3u8Download == null) {
synchronized (M3U8DownloadManager.class) {
if (m3u8Download == null) {
m3u8Download = new M3U8DownloadManager();
}
}
}
return m3u8Download;
}
/**
* 设置最大同时下载的数量
*
* @param maxDownloadTask
* @return
*/
public M3U8DownloadManager setMaxDownloadTask(int maxDownloadTask) {
this.maxDownloadTask = maxDownloadTask;
return this;
}
/**
* 初始化配置
*
* @return
*/
private M3U8DownloadManager initConfig() {
fixedThreadPool = Executors.newFixedThreadPool(maxDownloadTask);
return this;
}
/**
* 将任务加入线程池
*
* @param runnable
* @return
*/
public M3U8DownloadManager pushTask(M3U8SingleFileDownloadManager runnable) {
if (fixedThreadPool == null) {
initConfig();
}
fixedThreadPool.execute(runnable);
return this;
}
/**
* 停止全部任务
*/
public void stopAllTask() {
fixedThreadPool.shutdownNow();
fixedThreadPool = null;
}
}
5.（核心）编写单个m3u8视频下载的管理类（M3U8SingleFileDownloadManager）
public abstract class M3U8SingleFileDownloadManager implements Runnable {
//监听
public DownloadM3U8Listener listener;
private M3U8FileInfoModel downloadModel;
private int retryCount = 10;//重试次数
private boolean noAd = false;//无广告，true代表无广告，false代表有广告
private int maxDownloadCount = 50;//线程池大小
//自定义请求头
private Map requestHeaderMap = new HashMap<>();
private ExecutorService fixedThreadPool; //优化内存占用
private static final BlockingQueue BLOCKING_QUEUE = new LinkedBlockingQueue<>();
//已经下载的文件大小
private BigDecimal downloadBytes = new BigDecimal(0);
//已完成ts片段个数
private int finishedCount = 0;
//开始时间,-1表示从0开始
private int startTime = -1;
//结束时间，-1表示直到结尾
private int endTime = -1;
//解密后的片段，用于合并
private Set finishedFiles = new ConcurrentSkipListSet<>(Comparator.comparingInt(o -> Integer.parseInt(o.getName().replace(".xyz", ""))));
//是否是获取m3u8，如果是，那么将不执行下载
private boolean isGetM3U8 = false;
private boolean isSelectDown = false;
protected M3U8SingleFileDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd) {
this(url, dir, fileName, listener, 30, 100, noAd, false, -1, -1, new ArrayList<>());
}
protected M3U8SingleFileDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd, int startTime, int endTime) {
this(url, dir, fileName, listener, 30, 100, noAd, false, startTime, endTime, new ArrayList<>());
}
protected M3U8SingleFileDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd, boolean isGetM3U8) {
this(url, dir, fileName, listener, 30, 100, noAd, isGetM3U8, -1, -1, new ArrayList<>());
}
protected M3U8SingleFileDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd, ArrayList tsPartInfoModels) {
this(url, dir, fileName, listener, 30, 100, noAd, false, -1, -1, tsPartInfoModels);
}
private M3U8SingleFileDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener,
int retryCount, int maxDownloadCount, boolean noAd, boolean isGetM3U8, int startTime, int endTime, ArrayList tsPartInfoModels) {
downloadModel = new M3U8FileInfoModel();
this.listener = listener;
downloadModel.dir = dir;
downloadModel.fileName = fileName;
downloadModel.url = url;
downloadModel.tsPartInfoModels = tsPartInfoModels;
this.retryCount = retryCount;
this.noAd = noAd;
this.isGetM3U8 = isGetM3U8;
this.startTime = startTime;
this.endTime = endTime;
setThreadCount(maxDownloadCount);
requestHeaderMap.put("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36");
}
/**
* 设置线程数量
*
* @param maxDownloadCount
*/
public void setThreadCount(int maxDownloadCount) {
if (BLOCKING_QUEUE.size() < maxDownloadCount) {
for (int i = BLOCKING_QUEUE.size(); i < maxDownloadCount * Constant.FACTOR; i++) {
try {
BLOCKING_QUEUE.put(new byte[Constant.BYTE_COUNT]);
} catch (InterruptedException ignored) {
}
}
}
this.maxDownloadCount = maxDownloadCount;
}
@Override
public void run() {
boolean isSuccess = false;
try {
if (isGetM3U8) {
//校验字段
checkM3U8Field();
//获取真正下载的url和m3u8内容
getTrueDownloadUrlAndContent(false);
//将m3u8内容放进数据库
saveM3U8InfoToDb(downloadModel.fileName, downloadModel.tsPartInfoModels);
} else {
//开始下载
this.listener.startDownload(downloadModel, noAd);
//校验字段
checkField();
if (downloadModel.tsPartInfoModels != null && downloadModel.tsPartInfoModels.size() > 0) {
//说明是指定片段下载
isSelectDown = true;
} else {
//如果是片段就需要去拿m3u8数据
if (startTime != -1 || endTime != -1) {
//片段下载,m3u8信息不入库
//获取真正下载的url和m3u8内容
getTrueDownloadUrlAndContent(true);
} else {
//全量下载
if (hasDbData(downloadModel.fileName)) {
downloadModel.tsPartInfoModels = getM3U8InfoByName(downloadModel.fileName);
} else {
//获取真正下载的url和m3u8内容
getTrueDownloadUrlAndContent(false);
//将m3u8内容放进数据库
saveM3U8InfoToDb(downloadModel.fileName, downloadModel.tsPartInfoModels);
}
}
}
// 下载文件
downloadTsFile();
//开始合并视频
this.listener.mergeStart(downloadModel, noAd);
mergeTs();
this.listener.mergeEnd(downloadModel, noAd);
// 删除多余的ts片段
this.listener.deleteStart(downloadModel, noAd);
deleteFiles();
this.listener.deleteEnd(downloadModel, noAd);
}
isSuccess = true;
} catch (Exception e) {
e.printStackTrace();
this.listener.error(downloadModel, e, noAd);
} finally {
this.listener.complate(downloadModel, isSuccess, totalSize, finishedCount, noAd);
}
}
private int totalSize = 0;
/**
* 下载ts文件
*/
private void downloadTsFile() {
fixedThreadPool = Executors.newFixedThreadPool(maxDownloadCount);
int i = 0;
//如果生成目录不存在，则创建
File file1 = new File(downloadModel.dir);
if (!file1.exists())
file1.mkdirs();
//将任务加入线程池
for (int j = 0; j < downloadModel.tsPartInfoModels.size(); j++) {
M3U8TsPartInfoModel partInfoModel = downloadModel.tsPartInfoModels.get(j);
if (partInfoModel != null && partInfoModel.tsSet != null && partInfoModel.tsSet.size() > 0) {
for (M3U8TsPartInfoModel.TsInfoModel tsModel : partInfoModel.tsSet) {
i++;
fixedThreadPool.execute(getThread(tsModel.tsUrl, i, partInfoModel.method, partInfoModel.key, partInfoModel.iv, partInfoModel.isByte, partInfoModel.keyBytes));
}
}
}
fixedThreadPool.shutdown();
totalSize = i;
int consume = 0;
while (!fixedThreadPool.isTerminated()) {
try {
consume++;
BigDecimal bigDecimal = new BigDecimal(downloadBytes.toString());
Thread.sleep(1000L);
listener.process(consume,
downloadModel.url,
finishedCount,
totalSize,
new BigDecimal(finishedCount)
.divide(new BigDecimal(i), 4, BigDecimal.ROUND_HALF_UP)
.multiply(new BigDecimal(100))
.setScale(2, BigDecimal.ROUND_HALF_UP)
.floatValue(),
StringUtils.convertToDownloadSpeed(new BigDecimal(downloadBytes.toString()).subtract(bigDecimal), 3) + "/s",
noAd);
} catch (Exception e) {
e.printStackTrace();
}
}
}
/**
* 开启下载线程
*
* @param urls ts片段链接
* @param i ts片段序号
* @return 线程
*/
private Thread getThread(final String urls, final int i, final String method,
final String key, final String iv, boolean isByte, byte[] keyBytes) {
return new Thread() {
@Override
public void run() {
int count = 1;
HttpURLConnection httpURLConnection = null;
//xy为未解密的ts片段，如果存在，则删除
File file2 = new File(downloadModel.dir + FILESEPARATOR + i + ".xy");
if (file2.exists())
file2.delete();
OutputStream outputStream = null;
InputStream inputStream1 = null;
FileOutputStream outputStream1 = null;
byte[] bytes;
try {
bytes = BLOCKING_QUEUE.take();
} catch (InterruptedException e) {
bytes = new byte[Constant.BYTE_COUNT];
}
//重试次数判断
while (count <= retryCount) {
try {
//模拟http请求获取ts片段文件
URL url = new URL(urls);
httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setConnectTimeout((int) 20000L);
for (Map.Entry entry : requestHeaderMap.entrySet())
httpURLConnection.addRequestProperty(entry.getKey(), entry.getValue().toString());
httpURLConnection.setUseCaches(false);
httpURLConnection.setReadTimeout((int) 20000L);
httpURLConnection.setDoInput(true);
InputStream inputStream = httpURLConnection.getInputStream();
try {
outputStream = new FileOutputStream(file2);
} catch (FileNotFoundException e) {
e.printStackTrace();
continue;
}
int len;
//将未解密的ts片段写入文件
while ((len = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, len);
synchronized (this) {
downloadBytes = downloadBytes.add(new BigDecimal(len));
}
}
outputStream.flush();
inputStream.close();
inputStream1 = new FileInputStream(file2);
int available = inputStream1.available();
if (bytes.length < available)
bytes = new byte[available];
inputStream1.read(bytes);
File file = new File(downloadModel.dir + FILESEPARATOR + i + ".xyz");
outputStream1 = new FileOutputStream(file);
//去广告版
if ("NONE".equals(method) && noAd) {
break;
}
//开始解密ts片段，这里我们把ts后缀改为了xyz，改不改都一样
if ("NONE".equals(method) || "".equals(method)) {
SystemLogUtil.printSysDebugLog("解密", "NONE或者正常");
// 这里是一个m3u8文件出现多个解密方式，需要区别
outputStream1.write(bytes, 0, available);
} else {
byte[] decrypt = decrypt(file2.getName(), bytes, available, key, iv, method, isByte, keyBytes);
if (decrypt == null) {
SystemLogUtil.printSysDebugLog("解密", "正常");
outputStream1.write(bytes, 0, available);
} else {
SystemLogUtil.printSysDebugLog("解密", "AES" + isByte + " " + key + " keyBytes" + LwNetWorkTool.toHexString(keyBytes) + " " + file2.getName());
outputStream1.write(decrypt);
}
}
finishedFiles.add(file);
break;
} catch (Exception e) {
if (e instanceof InvalidKeyException || e instanceof InvalidAlgorithmParameterException) {
SystemLogUtil.printSysDebugLog("M3U8DOWNLOAD", "解密失败！");
break;
}
SystemLogUtil.printSysDebugLog("M3U8DOWNLOAD", e.getMessage().toString() + " 第" + count + "次获取链接重试！\t" + urls);
count++;
e.printStackTrace();
} finally {
try {
if (inputStream1 != null)
inputStream1.close();
if (outputStream1 != null)
outputStream1.close();
if (outputStream != null)
outputStream.close();
BLOCKING_QUEUE.put(bytes);
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
if (httpURLConnection != null) {
httpURLConnection.disconnect();
}
}
}
if (count > retryCount)
//自定义异常
throw new M3u8Exception("连接超时！");
finishedCount++;
SystemLogUtil.printSysDebugLog("M3U8DOWNLOAD", urls + "下载完毕！\t已完成" + finishedCount + "个，还剩" + (totalSize - finishedCount) + "个！");
}
};
}
/**
* 获取真正的m3u8内容的url以及要下载数据的models
*
* @param isTimePartDownload 是否是视频片段下载（true是部分内容下载，false表示全部内容下载）
*/
private void getTrueDownloadUrlAndContent(boolean isTimePartDownload) {
StringBuilder content = getUrlContent(downloadModel.url);
//判断是否是m3u8链接
String conStr = content.toString();
if (!conStr.contains("#EXTM3U"))
throw new M3u8Exception(downloadModel.url + "不是m3u8链接！");
//如果含有此字段，则说明ts片段链接需要从第二个m3u8链接获取
if (conStr.contains(".m3u8")) {
String[] split = conStr.split("\\n");
for (String s : split) {
if (s.contains(".m3u8")) {
//如果是url就需要拼接，如果带有http开头就直接赋值
if (StringUtils.isUrl(s)) {
downloadModel.trueUrl = s;
break;
}
//获取服务器域名部分
String relativeUrl = downloadModel.url.substring(0, downloadModel.url.lastIndexOf("/") + 1);
if (s.startsWith("/"))
s = s.replaceFirst("/", "");
downloadModel.trueUrl = mergeUrl(relativeUrl, s);
break;
}
}
getTsPartModels(isTimePartDownload);
} else {
downloadModel.trueUrl = downloadModel.url;
//初始化model
getTsPartModels(isTimePartDownload);
}
}
/**
* 获取TsPart数据
*
* @param isTimePartDownload 是否是视频片段下载（true是部分内容下载，false表示全部内容下载）
*/
private void getTsPartModels(boolean isTimePartDownload) {
SystemLogUtil.printSysDebugLog("M3U8SingleFile", downloadModel.toString());
//获取m3u8文本内容
StringBuilder content = getUrlContent(downloadModel.trueUrl);
//判断是否是m3u8链接
String conStr = content.toString();
if (!conStr.contains("#EXTM3U"))
throw new M3u8Exception(downloadModel.trueUrl + "不是m3u8链接！");
String relativeUrl = downloadModel.trueUrl.substring(0, downloadModel.trueUrl.lastIndexOf("/") + 1);
double curDuration = 0;
downloadModel.tsPartInfoModels = new ArrayList<>();
String[] split = conStr.split("\\n");
//判断是否有加密的部分
boolean hasKey = conStr.contains("#EXT-X-KEY");
M3U8TsPartInfoModel partInfoModel = null;
boolean isContent = false;
double duration = 0;
int index = 0;
if (!hasKey) {
partInfoModel = new M3U8TsPartInfoModel();
}
for (String s : split) {
//结束了
if (s.contains("#EXT-X-ENDLIST")) {
if (partInfoModel != null) {
//如果不为空，说明上一批已经有数据了，需要加入downloadModel
downloadModel.tsPartInfoModels.add(partInfoModel);
}
break;
}
if (hasKey) {
//获取key
if (s.contains("#EXT-X-KEY")) {
//获取密钥
if (partInfoModel != null) {
//如果不为空，说明上一批已经有数据了，需要加入downloadModel
downloadModel.tsPartInfoModels.add(partInfoModel);
}
//拆分密钥的详情
String[] split1 = s.split(",");
partInfoModel = new M3U8TsPartInfoModel();
index = 0;
for (String s1 : split1) {
if (s1.contains("METHOD")) {
partInfoModel.method = s1.split("=", 2)[1].replaceAll("\"", "");
continue;
}
if (s1.contains("URI")) {
partInfoModel.keyUrl = s1.split("=", 2)[1].replaceAll("\"", "");
continue;
}
if (s1.contains("IV")) {
partInfoModel.iv = s1.split("=", 2)[1].replaceAll("\"", "");
continue;
}
}
initKeyByUrl(partInfoModel, StringUtils.isUrl(partInfoModel.keyUrl) ? partInfoModel.keyUrl : mergeUrl(relativeUrl, partInfoModel.keyUrl));
isContent = false;
continue;
}
//获取内容
if (s.contains("#EXTINF")) {
isContent = true;
duration = Double.parseDouble(s.replaceAll("#EXTINF:", "").replaceAll(",", ""));
//统计
curDuration += duration;
continue;
}
if (curDuration < startTime
&& startTime != -1
&& isTimePartDownload) {
SystemLogUtil.printSysDebugLog("片段下载", "还未到时间" + curDuration + " " + startTime + " " + endTime);
continue;
}
setTsContent(relativeUrl, partInfoModel, isContent, s, duration, index);
if (curDuration > endTime
&& endTime != -1
&& isTimePartDownload) {
//超过了时间就退出
SystemLogUtil.printSysDebugLog("片段下载", "跳出循环" + curDuration + " " + startTime + " " + endTime);
if (partInfoModel != null) {
//如果不为空，说明上一批已经有数据了，需要加入downloadModel
downloadModel.tsPartInfoModels.add(partInfoModel);
}
break;
}
index++;
isContent = false;
} else {
//获取内容
if (s.contains("#EXTINF")) {
isContent = true;
duration = Double.parseDouble(s.replaceAll("#EXTINF:", "").replaceAll(",", ""));
//统计
curDuration += duration;
continue;
}
if (curDuration < startTime
&& startTime != -1
&& isTimePartDownload) {
SystemLogUtil.printSysDebugLog("片段下载", "还未到时间" + curDuration + " " + startTime + " " + endTime);
continue;
}
setTsContent(relativeUrl, partInfoModel, isContent, s, duration, index);
if (curDuration > endTime
&& endTime != -1
&& isTimePartDownload) {
//超过了时间就退出
SystemLogUtil.printSysDebugLog("片段下载", "跳出循环" + curDuration + " " + startTime + " " + endTime);
if (partInfoModel != null) {
//如果不为空，说明上一批已经有数据了，需要加入downloadModel
downloadModel.tsPartInfoModels.add(partInfoModel);
}
break;
}
index++;
isContent = false;
}
}
SystemLogUtil.printSysDebugLog("M3U8SingleFile", "获取文件部分：" + downloadModel.toString());
}
/**
* 设置ts文件内容
*
* @param relativeUrl
* @param partInfoModel
* @param isContent
* @param s
* @return
*/
private void setTsContent(String relativeUrl, M3U8TsPartInfoModel partInfoModel, boolean isContent, String s, double duration, int index) {
if (isContent) {
partInfoModel.addTsContent(StringUtils.isUrl(s) ? s : mergeUrl(relativeUrl, s), duration, index);
}
}
/**
* 去url获取key内容
*
* @param partInfoModel
* @param keyUrl
* @return
*/
private void initKeyByUrl(M3U8TsPartInfoModel partInfoModel, String keyUrl) {
if (!StringUtils.isEmpty(partInfoModel.method) && !"NONE".equals(partInfoModel.method)) {
int count = 1;
HttpURLConnection httpURLConnection = null;
while (count <= retryCount) {
try {
URL url = new URL(keyUrl);
httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setConnectTimeout((int) 10000L);
httpURLConnection.setReadTimeout((int) 10000L);
httpURLConnection.setUseCaches(false);
httpURLConnection.setDoInput(true);
for (Map.Entry entry : requestHeaderMap.entrySet())
httpURLConnection.addRequestProperty(entry.getKey(), entry.getValue().toString());
InputStream inputStream = httpURLConnection.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
byte[] bytes = new byte[128];
int len;
len = inputStream.read(bytes);
if (len == 1 << 4) {
partInfoModel.isByte = true;
partInfoModel.key = "isByte";
partInfoModel.keyBytes = Arrays.copyOf(bytes, 16);
} else {
partInfoModel.isByte = false;
partInfoModel.key = new String(Arrays.copyOf(bytes, len));
}
bufferedReader.close();
inputStream.close();
SystemLogUtil.printSysDebugLog("M3U8SingleFile", "获取key内容： ");
break;
} catch (Exception e) {
SystemLogUtil.printSysDebugLog("M3U8SingleFile", "获取key异常" + e.getMessage().toString() + " 第" + count + "获取链接重试！\t" + keyUrl);
count++;
e.printStackTrace();
} finally {
if (httpURLConnection != null) {
httpURLConnection.disconnect();
}
}
}
if (count > retryCount)
throw new M3u8Exception("获取key连接超时！");
}
}
/**
* 模拟http请求获取内容
*
* @param urls http链接
* @return 内容
*/
private StringBuilder getUrlContent(String urls) {
int count = 1;
HttpURLConnection httpURLConnection = null;
StringBuilder content = new StringBuilder();
while (count <= retryCount) {
try {
URL url = new URL(urls);
httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setConnectTimeout((int) 10000L);
httpURLConnection.setReadTimeout((int) 10000L);
httpURLConnection.setUseCaches(false);
httpURLConnection.setDoInput(true);
for (Map.Entry entry : requestHeaderMap.entrySet())
httpURLConnection.addRequestProperty(entry.getKey(), entry.getValue().toString());
String line;
InputStream inputStream = httpURLConnection.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
while ((line = bufferedReader.readLine()) != null)
content.append(line).append("\n");
bufferedReader.close();
inputStream.close();
SystemLogUtil.printSysDebugLog("M3U8SingleFile", "getUrlContent " + content.toString());
break;
} catch (Exception e) {
SystemLogUtil.printSysDebugLog("M3U8SingleFile", e.getMessage().toString() + " 第" + count + "获取链接重试！\t" + urls);
count++;
e.printStackTrace();
} finally {
if (httpURLConnection != null) {
httpURLConnection.disconnect();
}
}
}
if (count > retryCount)
throw new M3u8Exception("连接超时！");
return content;
}
/**
* 字段校验
*/
private void checkField() throws M3u8Exception {
if ("m3u8".compareTo(MediaFormat.getMediaFormat(downloadModel.url)) != 0)
throw new M3u8Exception(downloadModel.url + "不是一个完整m3u8链接！");
if (StringUtils.isEmpty(downloadModel.dir))
throw new M3u8Exception("视频存储目录不能为空！");
if (StringUtils.isEmpty(downloadModel.fileName))
throw new M3u8Exception("视频名称不能为空！");
finishedFiles.clear();
downloadBytes = new BigDecimal(0);
finishedCount = 0;
}
/**
* 字段校验
*/
private void checkM3U8Field() throws M3u8Exception {
if ("m3u8".compareTo(MediaFormat.getMediaFormat(downloadModel.url)) != 0)
throw new M3u8Exception(downloadModel.url + "不是一个完整m3u8链接！");
if (StringUtils.isEmpty(downloadModel.fileName))
throw new M3u8Exception("视频名称不能为空！");
finishedFiles.clear();
downloadBytes = new BigDecimal(0);
finishedCount = 0;
}
/**
* 解密ts
*
* @param name
* @param sSrc ts文件字节数组
* @param length
* @param sKey 密钥
* @return 解密后的字节数组
*/
private byte[] decrypt(String name, byte[] sSrc, int length, String sKey, String iv, String method, boolean isByte, byte[] keyBytes) throws Exception {
if (StringUtils.isNotEmpty(method) && !method.contains("AES"))
throw new M3u8Exception("未知的算法！" + method);
// 判断Key是否正确
if (StringUtils.isEmpty(sKey)) {
SystemLogUtil.printSysDebugLog("解密", name + " skey= " + sKey);
return null;
}
// 判断Key是否为16位
if (sKey.length() != 16 && !isByte) {
throw new M3u8Exception("Key长度不是16位！");
}
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS7Padding");
SecretKeySpec keySpec = new SecretKeySpec(isByte ? keyBytes : sKey.getBytes("UTF-8"), "AES");
byte[] ivByte;
if (iv.startsWith("0x"))
ivByte = StringUtils.hexStringToByteArray(iv.substring(2));
else ivByte = iv.getBytes();
if (ivByte.length != 16)
ivByte = new byte[16];
//如果m3u8有IV标签，那么IvParameterSpec构造函数就把IV标签后的内容转成字节数组传进去
AlgorithmParameterSpec paramSpec = new IvParameterSpec(ivByte);
cipher.init(Cipher.DECRYPT_MODE, keySpec, paramSpec);
return cipher.doFinal(sSrc, 0, length);
}
/**
* 组合url
*
* @param start
* @param end
* @return
*/
private String mergeUrl(String start, String end) {
if (end.startsWith("/"))
end = end.replaceFirst("/", "");
int position = 0;
String subEnd, tempEnd = end;
while ((position = end.indexOf("/", position)) != -1) {
subEnd = end.substring(0, position + 1);
if (start.endsWith(subEnd)) {
tempEnd = end.replaceFirst(subEnd, "");
break;
}
++position;
}
return start + tempEnd;
}
public static final SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss");
/**
* 合并文件
*/
private void mergeTs() {
try {
String saveName = downloadModel.fileName;
if (isSelectDown) {
saveName = downloadModel.fileName + sdf.format(new Date());
} else {
if (startTime == -1 && endTime == -1) {
//说明是下载完整的视频
saveName = downloadModel.fileName;
} else if (startTime != -1 && endTime != -1) {
saveName = "(" + StringUtils.timeParseBySecond(startTime) + "-" + StringUtils.timeParseBySecond(endTime) + ")" + downloadModel.fileName;
} else if (startTime == -1) {
saveName = "(00:00-" + StringUtils.timeParseBySecond(endTime) + ")" + downloadModel.fileName;
} else if (endTime == -1) {
saveName = "(" + StringUtils.timeParseBySecond(startTime) + "toEnd)" + downloadModel.fileName;
}
}
File file = new File(downloadModel.dir + FILESEPARATOR
+ saveName + ".mp4");
System.gc();
if (file.exists())
file.delete();
else file.createNewFile();
FileOutputStream fileOutputStream = new FileOutputStream(file);
byte[] b = new byte[4096];
for (File f : finishedFiles) {
SystemLogUtil.printSysDebugLog("合并文件", "合并了 " + f.getName());
FileInputStream fileInputStream = new FileInputStream(f);
int len;
while ((len = fileInputStream.read(b)) != -1) {
fileOutputStream.write(b, 0, len);
}
fileInputStream.close();
fileOutputStream.flush();
}
fileOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* 合并完成后删除ts文件
*/
private void deleteFiles() {
File file = new File(downloadModel.dir);
for (File f : file.listFiles()) {
if (f.getName().endsWith(".xy") || f.getName().endsWith(".xyz"))
f.delete();
}
}
/**
* 判断是否有数据
*
* @param fileName
* @return
*/
protected abstract boolean hasDbData(String fileName);
/**
* 获取
*
* @param fileName
* @return
*/
protected abstract ArrayList getM3U8InfoByName(String fileName) throws Exception;
/**
* 保存m3u8信息
*
* @param fileName
* @param tsPartInfoModels
*/
protected abstract void saveM3U8InfoToDb(String fileName, ArrayList tsPartInfoModels);
}
至此，下载的库就封装完成了，接下来我们看看怎么样使用这个下载库。
参考下面的帮助类：
/**
*
* author : lawwing
* time : 2021年03月25日
* desc : 添加保存功能的
* version: 1.0
*
*/
public class M3U8SaveDbAndDownloadManager extends M3U8SingleFileDownloadManager {
public M3U8SaveDbAndDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd) {
super(url, dir, fileName, listener, noAd);
}
public M3U8SaveDbAndDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd, int startTime, int endTime) {
super(url, dir, fileName, listener, noAd, startTime, endTime);
}
public M3U8SaveDbAndDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd, ArrayList tsPartInfoModels) {
super(url, dir, fileName, listener, noAd, tsPartInfoModels);
}
public M3U8SaveDbAndDownloadManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd, boolean isGetM3U8) {
super(url, dir, fileName, listener, noAd, isGetM3U8);
}
/**
* 获取m3u8内容的manager
*
* @param url
* @param fileName
* @param listener
* @return
*/
public static M3U8SingleFileDownloadManager getFindM3U8DataManager(String url, String fileName, DownloadM3U8Listener listener) {
return new M3U8SaveDbAndDownloadManager(url, "", fileName, listener, false, true);
}
/**
* 获取下载任务的manager
*
* @param url
* @param fileName
* @param listener
* @return
*/
public static M3U8SingleFileDownloadManager getDownloadDataManager(String url, String dir, String fileName, DownloadM3U8Listener listener, boolean noAd) {
return new M3U8SaveDbAndDownloadManager(url, dir, fileName, listener, noAd);
}
/**
* 获取下载指定时间任务的manager
*
* @param url
* @param dir
* @param fileName
* @param listener
* @param noAd
* @param startTime
* @param endTime
* @return
*/
public static M3U8SingleFileDownloadManager getDownloadDataByTimeManager(String url, String dir, String fileName,
DownloadM3U8Listener listener, boolean noAd,
Integer startTime, Integer endTime) {
return new M3U8SaveDbAndDownloadManager(url, dir, fileName, listener, noAd, startTime.intValue(), endTime.intValue());
}
/**
* 获取下载指定任务片的manager
*
* @param url
* @param dir
* @param fileName
* @param listener
* @param noAd
* @return
*/
public static M3U8SingleFileDownloadManager getDownloadDataSelectManager(String url, String dir, String fileName,
DownloadM3U8Listener listener, boolean noAd, ArrayList tsPartInfoModels) {
return new M3U8SaveDbAndDownloadManager(url, dir, fileName, listener, noAd, tsPartInfoModels);
}
@Override
protected boolean hasDbData(String fileName) {
try {
int count = VideoDatabase.Companion.getDBInstace().getM3U8PartDao().checkCountByName(fileName);
if (count > 0) {
return true;
} else {
return false;
}
} catch (Exception e) {
e.printStackTrace();
}
return false;
}
@Override
protected ArrayList getM3U8InfoByName(String fileName) throws Exception {
ArrayList result = new ArrayList<>();
List datasByNameAsc = VideoDatabase.Companion.getDBInstace().getM3U8PartDao().findDatasByNameAsc(fileName);
if (datasByNameAsc != null && datasByNameAsc.size() > 0) {
for (int i = 0; i < datasByNameAsc.size(); i++) {
M3U8FilePartDbInfo m3U8FilePartDbInfo = datasByNameAsc.get(i);
if (m3U8FilePartDbInfo != null) {
int m3u8id = m3U8FilePartDbInfo.getId();
M3U8TsPartInfoModel partModel = new M3U8TsPartInfoModel();
partModel.method = m3U8FilePartDbInfo.getMethod();
partModel.key = m3U8FilePartDbInfo.getKey();
partModel.keyUrl = m3U8FilePartDbInfo.getKeyUrl();
partModel.keyBytes = m3U8FilePartDbInfo.getKeyBytes();
partModel.iv = m3U8FilePartDbInfo.getIv();
partModel.isByte = m3U8FilePartDbInfo.isByte();
List tsFileDbInfoList = VideoDatabase.Companion.getDBInstace().getTsFileDao().findDatasByPartIdAsc(m3u8id);
if (tsFileDbInfoList != null && tsFileDbInfoList.size() > 0) {
for (int j = 0; j < tsFileDbInfoList.size(); j++) {
TsFileDbInfo tsFileDbInfo = tsFileDbInfoList.get(j);
partModel.tsSet.add(new M3U8TsPartInfoModel.TsInfoModel(tsFileDbInfo.getTsPath(),
tsFileDbInfo.getDuration(),
tsFileDbInfo.getPartindex()));
}
}
result.add(partModel);
}
}
}
return result;
}
@Override
public void saveM3U8InfoToDb(String fileName, ArrayList tsPartInfoModels) {
//如果已经有了，就删除数据库之前的数据
List datasByNameAsc = VideoDatabase.Companion.getDBInstace().getM3U8PartDao().findDatasByNameAsc(fileName);
if (datasByNameAsc != null && datasByNameAsc.size() > 0) {
for (int i = 0; i < datasByNameAsc.size(); i++) {
M3U8FilePartDbInfo m3U8FilePartDbInfo = datasByNameAsc.get(i);
VideoDatabase.Companion.getDBInstace().getTsFileDao().deleteByPartId(m3U8FilePartDbInfo.getId());
VideoDatabase.Companion.getDBInstace().getM3U8PartDao().deleteById(m3U8FilePartDbInfo.getId());
}
}
// 保存到数据库
if (tsPartInfoModels != null && tsPartInfoModels.size() > 0) {
int partIndex = 0;
for (int j = 0; j < tsPartInfoModels.size(); j++) {
M3U8TsPartInfoModel partInfoModel = tsPartInfoModels.get(j);
M3U8FilePartDbInfo dbPartInfo = new M3U8FilePartDbInfo();
dbPartInfo.setPartindex(partIndex);
dbPartInfo.setName(fileName);
dbPartInfo.setKey(partInfoModel.key);
dbPartInfo.setKeyUrl(partInfoModel.keyUrl);
dbPartInfo.setKeyBytes(partInfoModel.keyBytes);
dbPartInfo.setByte(partInfoModel.isByte);
dbPartInfo.setIv(partInfoModel.iv);
dbPartInfo.setMethod(partInfoModel.method);
//先插入数据再获取到id
VideoDatabase.Companion.getDBInstace().getM3U8PartDao().insert(dbPartInfo);
int partId = VideoDatabase.Companion.getDBInstace().getM3U8PartDao().findPartIdByNameMaxPartindex(fileName);
ArrayList tsList = new ArrayList<>();
if (partInfoModel != null && partInfoModel.tsSet != null && partInfoModel.tsSet.size() > 0) {
for (M3U8TsPartInfoModel.TsInfoModel tsInfo : partInfoModel.tsSet) {
TsFileDbInfo tsModel = new TsFileDbInfo();
tsModel.setM3u8id(partId);
tsModel.setTsPath(tsInfo.tsUrl);
tsModel.setDuration(tsInfo.duration);
tsModel.setPartindex(tsInfo.index);
tsList.add(tsModel);
}
}
VideoDatabase.Companion.getDBInstace().getTsFileDao().insertAll(tsList);
partIndex++;
}
}
}
}
由于将m3u8信息保存到数据库中，可减少服务器请求次数，防止ip被封禁，所以下面附上数据库的建表相关代码（使用了room数据库框架）：
@Entity(tableName = "M3U8FilePartInfo")
class M3U8FilePartDbInfo : BaseObservable() {
@PrimaryKey(autoGenerate = true)
@ColumnInfo(name = "id")
var id: Int? = null
/**
* 这个块所在的位置
*/
@ColumnInfo(name = "partindex")
var partindex: Int? = null
/**
* 名字
*/
@ColumnInfo(name = "name")
var name: String? = null
/**
*key的地址
*/
@ColumnInfo(name = "keyUrl")
var keyUrl: String? = ""
/**
*加密方法
*/
@ColumnInfo(name = "method")
var method: String? = ""
/**
*iv
*/
@ColumnInfo(name = "iv")
var iv: String? = ""
/**
*key
*/
@ColumnInfo(name = "key")
var key: String? = ""
/**
*密钥字节
*/
@ColumnInfo(name = "keyBytes")
var keyBytes: ByteArray? = ByteArray(16)
/**
*key是否为字节
*/
@ColumnInfo(name = "isByte")
var isByte: Boolean? = false
}
@Entity(tableName = "TsFileInfo")
class TsFileDbInfo : BaseObservable() {
@PrimaryKey(autoGenerate = true)
@ColumnInfo(name = "id")
var id: Int? = null
/**
* 关联的表M3U8FilePartInfo的id
*/
@ColumnInfo(name = "m3u8id")
var m3u8id: Int? = null
/**
* 这个ts块所在的位置
*/
@ColumnInfo(name = "partindex")
var partindex: Int? = null
/**
* 分片路径
*/
@ColumnInfo(name = "tsPath")
var tsPath: String? = null
/**
* 分片时长
*/
@ColumnInfo(name = "duration")
var duration: Double? = null
}
网上的轮子纵然好用，但是也不会完全适合自己的需求，所以还得去修改，去创新。
自己封装的框架不仅可以下载完整的m3u8视频的文件，还可以选择分片，选择指定时间去下载所需要的视频片段。除此之外，研究了该网站的视频，发现广告都是插在不加密的视频部分，所以不加密的视频不下载，就成了去除广告的功能了。