最全的CSE-CIC-IDS2018 下载

目录

前言

下载

逐条下载：

命令行解读：

递归下载

---

参考链接：https://blog.csdn.net/u013698706/article/details/106422480

前言

复现论文，CSE-CIC-IDS2018全部有452.8GiB，所幸有处理好的CSV文件，2018的合计有6GiB左右；

四处找寻不到，CSDN会员资源下载下来里面是不对的内容，没办法自己去官网下载，过程不易，特总结一下方法，以及分享我下载的2018数据集。

我采用的下载办法，是用windows系统下的

下载

【1】 官网下载AWS工具
或者用我下载下来的(，windows64位)
【2】 在aws的安装文件夹中打开cmd命令窗口，运行：

aws s3 ls --no-sign-request "s3://cse-cic-ids2018" --recursive --human-readable --summarize

结果显示的是文件的一个总列表详细信息

C:\Users\LCF>aws s3 ls --no-sign-request "s3://cse-cic-ids2018" --recursive --human-readable --summarize
D:\Program Files\Amazon\AWSCLI\.\dateutil\parser\_parser.py:1189: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
2018-10-10 19:52:09    0 Bytes Original Network Traffic and Log data/
2018-10-10 19:52:23    0 Bytes Original Network Traffic and Log data/Friday-02-03-2018/
2018-10-10 20:00:39  225.8 MiB Original Network Traffic and Log data/Friday-02-03-2018/logs.zip
2018-10-10 20:00:51   41.7 GiB Original Network Traffic and Log data/Friday-02-03-2018/pcap.zip
2018-10-10 19:52:34    0 Bytes Original Network Traffic and Log data/Friday-16-02-2018/
2018-10-10 20:45:49  148.1 MiB Original Network Traffic and Log data/Friday-16-02-2018/logs.zip
2018-10-10 20:46:01   35.9 GiB Original Network Traffic and Log data/Friday-16-02-2018/pcap.zip
2018-10-10 19:52:41    0 Bytes Original Network Traffic and Log data/Friday-23-02-2018/
2018-10-10 20:46:10  199.8 MiB Original Network Traffic and Log data/Friday-23-02-2018/logs.zip
2018-10-10 20:46:31   55.0 GiB Original Network Traffic and Log data/Friday-23-02-2018/pcap.zip
2018-10-10 19:52:47    0 Bytes Original Network Traffic and Log data/Thursday-01-03-2018/
2018-10-10 21:41:13  217.1 MiB Original Network Traffic and Log data/Thursday-01-03-2018/logs.zip
2018-10-10 21:41:45   48.8 GiB Original Network Traffic and Log data/Thursday-01-03-2018/pcap.zip
2018-10-10 19:52:54    0 Bytes Original Network Traffic and Log data/Thursday-15-02-2018/
2018-10-10 21:41:28  142.6 MiB Original Network Traffic and Log data/Thursday-15-02-2018/logs.zip
2018-10-10 21:41:55   38.4 GiB Original Network Traffic and Log data/Thursday-15-02-2018/pcap.zip
2018-10-10 19:53:01    0 Bytes Original Network Traffic and Log data/Thursday-22-02-2018/
2018-10-10 21:41:42  195.3 MiB Original Network Traffic and Log data/Thursday-22-02-2018/logs.zip
2018-10-10 21:42:27   46.8 GiB Original Network Traffic and Log data/Thursday-22-02-2018/pcap.zip
2018-10-10 19:53:07    0 Bytes Original Network Traffic and Log data/Tuesday-20-02-2018/
2018-10-10 22:39:45  178.9 MiB Original Network Traffic and Log data/Tuesday-20-02-2018/logs.zip
2018-10-10 22:40:40   41.3 GiB Original Network Traffic and Log data/Tuesday-20-02-2018/pcap.rar
2018-10-10 19:53:14    0 Bytes Original Network Traffic and Log data/Wednesday-14-02-2018/
2018-10-11 00:44:20  133.7 MiB Original Network Traffic and Log data/Wednesday-14-02-2018/logs.zip
2018-10-11 20:22:03   37.2 GiB Original Network Traffic and Log data/Wednesday-14-02-2018/pcap.zip
2018-10-10 19:53:21    0 Bytes Original Network Traffic and Log data/Wednesday-21-02-2018/
2018-10-11 00:44:34  185.6 MiB Original Network Traffic and Log data/Wednesday-21-02-2018/logs.zip
2018-10-11 21:35:15   49.8 GiB Original Network Traffic and Log data/Wednesday-21-02-2018/pcap.zip
2018-10-10 19:53:28    0 Bytes Original Network Traffic and Log data/Wednesday-28-02-2018/
2018-10-11 00:44:47  216.1 MiB Original Network Traffic and Log data/Wednesday-28-02-2018/logs.zip
2018-10-11 22:21:03   49.6 GiB Original Network Traffic and Log data/Wednesday-28-02-2018/pcap.zip
2018-10-12 00:02:25    0 Bytes Processed Traffic Data for ML Algorithms/
2018-10-12 00:02:49  336.0 MiB Processed Traffic Data for ML Algorithms/Friday-02-03-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:10  318.3 MiB Processed Traffic Data for ML Algorithms/Friday-16-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:33  365.1 MiB Processed Traffic Data for ML Algorithms/Friday-23-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:59    3.8 GiB Processed Traffic Data for ML Algorithms/Thuesday-20-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:08:38  102.8 MiB Processed Traffic Data for ML Algorithms/Thursday-01-03-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:08:48  358.5 MiB Processed Traffic Data for ML Algorithms/Thursday-15-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:09:20  364.9 MiB Processed Traffic Data for ML Algorithms/Thursday-22-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:09:44  341.6 MiB Processed Traffic Data for ML Algorithms/Wednesday-14-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:10:12  313.7 MiB Processed Traffic Data for ML Algorithms/Wednesday-21-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:10:33  199.6 MiB Processed Traffic Data for ML Algorithms/Wednesday-28-02-2018_TrafficForML_CICFlowMeter.csv

Total Objects: 42
   Total Size: 452.8 GiB

 

【3】我们需要下载的是最后十个CSV文件，经过测试，逐条下载比递归下载要快。举例：

逐条下载：

需要逐个替换下载的文件

aws s3 cp --no-sign-request "s3://cse-cic-ids2018/Processed Traffic Data for ML Algorithms/Wednesday-28-02-2018_TrafficForML_CICFlowMeter.csv" D:\A_Datasets\IDS2018\Wednesday-28-02-2018_TrafficForML_CICFlowMeter.csv

命令行解读：

代表链接文件：

 "s3://cse-cic-ids2018/Processed Traffic Data for ML Algorithms/Wednesday-28-02-2018_TrafficForML_CICFlowMeter.csv" 

代表我的存储路径和命名 

D:\A_Datasets\IDS2018\Wednesday-28-02-2018_TrafficForML_CICFlowMeter.csv

递归下载

aws s3 cp --no-sign-request "s3://cse-cic-ids2018/Processed Traffic Data for ML Algorithms/" D:\A_Datasets\IDS2018 --recursive

我的下载：