CSE-CIC-IDS2018 下载

个人博客文章：CSE-CIC-IDS2018下载

在researchgate看到这位哥们遇到了跟我一样的问题： UNB官方 给的资源超过 200 GiB，但是我只想获取其经过处理后的 .csv 文件。

下面来自网友的解决方案！

一、安装 aws cli

1 macOS

官方：https://docs.aws.amazon.com/zh_cn/cli/latest/userguide/install-cliv2-mac.html

$ curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg" 
$ sudo installer -pkg AWSCLIV2.pkg -target /

• curl 方式下载可能会很慢，可以拷贝里面的 url 到 bt 工具进行下载。

安装后，通过$ aws --version 确认安装。

2 Ubuntu

$ apt-get install -Y aws

二 、列出 aws 资源

总共 452.8 GiB，找到我们需要下载的资源，给出指定的资源路径。

$ aws s3 ls --no-sign-request "s3://cse-cic-ids2018" --recursive --human-readable --summarize

2018-10-10 19:52:09    0 Bytes Original Network Traffic and Log data/
2018-10-10 19:52:23    0 Bytes Original Network Traffic and Log data/Friday-02-03-2018/
2018-10-10 20:00:39  225.8 MiB Original Network Traffic and Log data/Friday-02-03-2018/logs.zip
2018-10-10 20:00:51   41.7 GiB Original Network Traffic and Log data/Friday-02-03-2018/pcap.zip
2018-10-10 19:52:34    0 Bytes Original Network Traffic and Log data/Friday-16-02-2018/
2018-10-10 20:45:49  148.1 MiB Original Network Traffic and Log data/Friday-16-02-2018/logs.zip
2018-10-10 20:46:01   35.9 GiB Original Network Traffic and Log data/Friday-16-02-2018/pcap.zip
2018-10-10 19:52:41    0 Bytes Original Network Traffic and Log data/Friday-23-02-2018/
2018-10-10 20:46:10  199.8 MiB Original Network Traffic and Log data/Friday-23-02-2018/logs.zip
2018-10-10 20:46:31   55.0 GiB Original Network Traffic and Log data/Friday-23-02-2018/pcap.zip
2018-10-10 19:52:47    0 Bytes Original Network Traffic and Log data/Thursday-01-03-2018/
2018-10-10 21:41:13  217.1 MiB Original Network Traffic and Log data/Thursday-01-03-2018/logs.zip
2018-10-10 21:41:45   48.8 GiB Original Network Traffic and Log data/Thursday-01-03-2018/pcap.zip
2018-10-10 19:52:54    0 Bytes Original Network Traffic and Log data/Thursday-15-02-2018/
2018-10-10 21:41:28  142.6 MiB Original Network Traffic and Log data/Thursday-15-02-2018/logs.zip
2018-10-10 21:41:55   38.4 GiB Original Network Traffic and Log data/Thursday-15-02-2018/pcap.zip
2018-10-10 19:53:01    0 Bytes Original Network Traffic and Log data/Thursday-22-02-2018/
2018-10-10 21:41:42  195.3 MiB Original Network Traffic and Log data/Thursday-22-02-2018/logs.zip
2018-10-10 21:42:27   46.8 GiB Original Network Traffic and Log data/Thursday-22-02-2018/pcap.zip
2018-10-10 19:53:07    0 Bytes Original Network Traffic and Log data/Tuesday-20-02-2018/
2018-10-10 22:39:45  178.9 MiB Original Network Traffic and Log data/Tuesday-20-02-2018/logs.zip
2018-10-10 22:40:40   41.3 GiB Original Network Traffic and Log data/Tuesday-20-02-2018/pcap.rar
2018-10-10 19:53:14    0 Bytes Original Network Traffic and Log data/Wednesday-14-02-2018/
2018-10-11 00:44:20  133.7 MiB Original Network Traffic and Log data/Wednesday-14-02-2018/logs.zip
2018-10-11 20:22:03   37.2 GiB Original Network Traffic and Log data/Wednesday-14-02-2018/pcap.zip
2018-10-10 19:53:21    0 Bytes Original Network Traffic and Log data/Wednesday-21-02-2018/
2018-10-11 00:44:34  185.6 MiB Original Network Traffic and Log data/Wednesday-21-02-2018/logs.zip
2018-10-11 21:35:15   49.8 GiB Original Network Traffic and Log data/Wednesday-21-02-2018/pcap.zip
2018-10-10 19:53:28    0 Bytes Original Network Traffic and Log data/Wednesday-28-02-2018/
2018-10-11 00:44:47  216.1 MiB Original Network Traffic and Log data/Wednesday-28-02-2018/logs.zip
2018-10-11 22:21:03   49.6 GiB Original Network Traffic and Log data/Wednesday-28-02-2018/pcap.zip
2018-10-12 00:02:25    0 Bytes Processed Traffic Data for ML Algorithms/
2018-10-12 00:02:49  336.0 MiB Processed Traffic Data for ML Algorithms/Friday-02-03-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:10  318.3 MiB Processed Traffic Data for ML Algorithms/Friday-16-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:33  365.1 MiB Processed Traffic Data for ML Algorithms/Friday-23-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:59    3.8 GiB Processed Traffic Data for ML Algorithms/Thuesday-20-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:08:38  102.8 MiB Processed Traffic Data for ML Algorithms/Thursday-01-03-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:08:48  358.5 MiB Processed Traffic Data for ML Algorithms/Thursday-15-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:09:20  364.9 MiB Processed Traffic Data for ML Algorithms/Thursday-22-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:09:44  341.6 MiB Processed Traffic Data for ML Algorithms/Wednesday-14-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:10:12  313.7 MiB Processed Traffic Data for ML Algorithms/Wednesday-21-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:10:33  199.6 MiB Processed Traffic Data for ML Algorithms/Wednesday-28-02-2018_TrafficForML_CICFlowMeter.csv

Total Objects: 42
   Total Size: 452.8 GiB

三、下载 CSV 数据集

aws zones : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

$ aws s3 cp --no-sign-request "s3://cse-cic-ids2018/Processed Traffic Data for ML Algorithms/" cicids2018 --recursive

四、Tips

aws 下载可能很慢，可以使用下面的方式获取

• 使用 VPS

~

• 墙

~

• Colab 保存至 Google Drive

~