Github搜索
总结来说：
secrets are committed often, and are discoverable very quickly, likely before the affected parties have time to react. Attackers can, and have, used similar techniques to identify secrets and use them for malicious purposes.
虽然这些github数据集历史数据对于衡量问题的规模或确定一段时间内的趋势非常有用，但是大多数组织来说，对于未来如何监视或防止新的secrets泄露更加感兴趣。
优秀的思路参考
https://lightless.me/archives/How-To-Designing-A-Faster-Than-Faster-GitHub-Monitoring-System.html
其代码实现：
https://github.com/lightless233/geye
可以对照这个项目参考：
https://github.com/VKSRC/Github-Monitor
如何防止
如果你对开发环境可控
因为git有一个hook脚本的功能，可以在commit之前进行一些检查，如果你对开发的git环境完全可控，完全可以进行hook，在敏感信息commit之前就拦截下来。
如果你对git仓库完全可控
由于github支持webhook，你可以监控一些commit/push事件，然后进行hook，一旦发生，就进行扫描行为。
https://developer.github.com/webhooks/
如果完全移除某个repo：
https://help.github.com/en/articles/removing-sensitive-data-from-a-repository
参考
https://duo.com/labs/research/how-to-monitor-github-for-secrets
https://www.ndss-symposium.org/ndss-paper/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories/
Github数据集
https://www.gharchive.org/
Github自己上传的：
https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset&id=46ee22ab-2ca4-4750-81a7-3ee0f0150dcb
细节
github敏感信息泄露研究：
https://www.ndss-symposium.org/ndss-paper/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories/
paper：
https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04B-3_Meli_paper.pdf
video：
https://www.youtube.com/watch?v=N-pg_47s5Ok&index=4&t=1s&list=PLfUWWM-POgQtjEA_FIN7s0XFWoRdW4lil
github数据集：
https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset&q=github&id=46ee22ab-2ca4-4750-81a7-3ee0f0150dcb&pli=1
查询查看界面：
https://console.cloud.google.com/bigquery?organizationId=&angularJsUrl=%2Fbigquery%3Fp%3Dbigquery-public-data%26d%3Dgithub_repos%26page%3Ddataset%26organizationId%3D%26creatingProject%3Dtrue%26angularJsUrl%3D%252Fbigquery%253Fp%253Dbigquery-public-data%2526d%253Dgithub_repos%2526page%253Ddataset%2526supportedpurview%253Dproject%2526organizationId%253D0%2526creatingProject%253Dtrue%26project%3Dfestive-idea-254209%26folder%3D%26supportedpurview%3Dproject&project=festive-idea-254209&folder=&supportedpurview=project&p=bigquery-public-data&d=github_repos&t=files&page=table
Github监控项目汇总
Audit git repos for secrets
使用Django的一套github监控框架
git-secrets脚本：
https://github.com/awslabs/git-secrets/blob/3958dacceeebeab84e2a3c686c00fb9bde17cb55/git-secrets
匹配正则汇总
https://github.com/zricethezav/gitleaks/blob/065b6216049d71e7f3c28dec3f4e93a24b304033/gitleaks.toml
https://github.com/michenriksen/gitrob/blob/7be4c5306a61383a3ba16777b520b3c2a8956a1e/core/signatures.go
https://github.com/dxa4481/truffleHog/blob/0d6f2dfea5f9e9b196414f3925b988e1ba62880f/scripts/searchOrg.py
https://github.com/eth0izzle/shhgit/blob/f9b4febcd6ec6c1d509b28efbad6dc1ca9d17837/config.yaml
https://github.com/BishopFox/GitGot/blob/3a754dfcf66707a68d7507aabb5cf44d48f5e924/checks/default.list
附录
代码搜索语法：
https://help.github.com/en/articles/searching-code
You must be signed in to search for code across all public repositories.
用户只能在登录状态下，搜索整个github的仓库的代码
Code in forks is only searchable if the fork has more stars than the parent repository. Forks with fewer stars than the parent repository are not indexed for code search.
forks中的代码，只有中fork的仓库的star比原仓库多时，才会被索引，否则不会。
Only the default branch is indexed for code search. In most cases, this will be the master branch.
默认只会搜索默认（通常为master）分支中的代码。
Only files smaller than 384 KB are searchable.
只有小于384 KB大小的文件才能被搜索。
Only repositories with fewer than 500,000 files are searchable.
只有文件个数少于500,000的仓库才能被搜索。
You can’t use the following wildcard characters as part of your search query: . , : ; / \ ` ’ " = * ! ? # $ & + ^ | ~ < > ( ) { } [ ]. The search will simply ignore these symbols.
以下特殊符号不能作为搜索关键词。
笔记
由于github不支持正则在线匹配，所以只能搜索特定关键词，然后离线匹配，这篇文章里说的也是这种方式：
https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04B-3_Meli_paper.pdf