Conference notes: The Bug Hunters Methodology v3(ish) (LevelUp 0x02 / 2018)

youtube上 bug hunters methodology的笔记转载别人的笔记做一下记录

回头我在慢慢整理成中文的吧好累

https://pentester.land/conference-notes/2018/08/02/levelup-2018-the-bug-hunters-methodology-v3.html

Hi, these are the notes I took while watching “The Bug Hunters Methodology v3(ish)” talk given by Jason Haddix on LevelUp 0x02 / 2018.

Links

Video

Slides

About

This talk is about Jason Haddix’s bug hunting methodology. It is an upgrade of:

The Bug Hunter’s Methodology AKA How to Shot Web (Defcon 23) (Slides)

The Bug Hunters Methodology v2.1 (Slides)

Light reading

The Web Application Hacker’s Handbook: Finding and Exploiting Security Flaws, 2nd Edition

OWASP Testing Guide v4

Web Hacking 101: How to Make Money Hacking Ethically

Breaking into Information Security: Learning the Ropes 101 (Free)

Mastering Modern Web Penetration Testing

Recon & Enumeration

Discovering IP Space

Goal: Given an org name, identify both their hosts/top-level domains & IP space

ASN’s

ASN = Autonomous System Number

http://bgp.he.net

Enter a company name or a keyword => ASNs listed, select 1 => IP ranges listed in Prefixes v4 tab

One of the only sites that support search by keyword (e.g. tesla, tesla motors, tesla inc…)

ARIN & RIPE

https://whois.arin.net/ui/query.do

https://apps.db.ripe.net/db-web-ui/#/fulltextsearch

keyword search to do whois lookups (e.g. Tesla Motors)

Rev WHOIS

https://reverse.report

Enter your bug bounty target’s a main domain (e.g. tesla.com) to get anything related via reverse lookup

Ed.: This site is down but there are alternatives: https://opendata.rapid7.com/sonar.rdns_v2/ & http://ipv4info.com/

Jason Haddix@Jhaddix

Sad day... what happened tohttp://reverse.report ?

11:05 AM - Aug 1, 2018

Twitter Ads info and privacy

See Jason Haddix's other Tweets

Twitter Ads info and privacy

Shodan Organization

https://www.shodan.io/search?query=org%3a%22tesla+motors%22

Search for org:tesla motors

Discovering New Targets (Brands & TLDs)

Goal: Find new brands & Top-Level Domains

Brand/TLD Discovery

Acquisitions

Wikipedia the org

Crunchbase acquisitions section

https://www.crunchbase.com/organization/tesla-motors/acquisitions

Linked discovery

Burp spidering

Demo (min 14.45)

Turn off passive scanning (it generates a lot of results, taking up a lot of memory because we’re gonna spider a lot of sites) - Scanner > Live Passive Scanning > Check Don’t scan

Set forms to auto submit (if feeling frisky)

Spider tab > Application login section > Check Handle as ordinary form

Spider tab > Form submission section > Automatically submit using the following rules…

If you submit regular form & there’s an email generating form, you would blast tesla

So use your discretion to configure these 2 forms. If it’s a small site with no email generating form, it’s OK to enable automatic forms submission

Set scope to advanced control & use string of target name (not a normal FQDN)

Goal: Show only links that have tesla in the URL in the Site map

Click yes for to “…stop sending out of scope items to the history…”

Show only in scope items in the Target / Site map

Manually walk + browse, then spider all hosts recursively

Spidering can take 5 to 25 min depending

profit (more targets)!

Allows finding Tesla domains hosted on third parties like https://teslaenergy.formstack.com

Weighted link and reverse tracker analysis

Domlink

by Vincent Yiu

Idea: Recursively looks at reverse whois programmatically based on who registered a domain, and then creates a link between those domains

Usage: python ./domLink.py -d vip.com -o vip.out.txt

Do a whois lookup on vip.com. Then from the WHOIS information, based on the registrar & other data, recursively look at any other WHOIS record that has the same information

Hasn’t worked well for Jason yet but he likes the idea

Builtwith

Idea: Links together the relationship of a site based on its analytics trackers (ie domains using the same analytics code)

Usage: Relationships tab, Related Domains section

Gives you a heat map of how each domain is related to your target

Helped him find sites that are related and in scope but not explicitly listed

Others

Google dork for:

the privacy policy link of the company

Discovering new targets (Subdomains)

Goal: Finding subdomains

Two main ways:

Subdomain scraping: Find subdomains referenced online somewhere

Subdomain brute forcing: Try to resolve subdomains based on a wordlist

Subdomain scraping

It’s all about sources

Probably 25 to 40 good sources. Examples:

Ask.com

crt.sh

Hacker target

https://dnsdumpster.com

Ptrarchive.com

DNSDB Search

Bing

Baidu

Yahoo!

Google

Robtex!

Censys

CertDB

Cert Spotter

Netcraft

Passive Total

F-secure Riddler

Dogpile

Security Trails

ThreatMiner

VirusTotal

Wayback Machine

ThreatCrowd

Tools you use must have the right sources and be executed relatively quickly

Jason used to use Sublist3r & Altdns but now prefers using only Amass & Subfinder

Amass

https://github.com/caffix/amass by @jeff_foley

Up to 22 sources

Includes Reverse DNS methods & permutation scanning (dev-1.netflix.com, dev-2.netflix.com)

Usage

root@Test2:~/tools/amass#cat amass.sh#!/bin/bashmkdir $1touch $1/$1.txtamass -active -d $1 |tee /root/tools/amass/$1/$1.txtroot@Test2:~/tools/amass#./amass.sh netflix.com

Subfinder

https://github.com/ice3man543/subfinder by Iceman

Most sources included in Amass

But also include Json output & a multi resolver for bruteforce…

Idea: Integrate scraping & bruteforcing in a single subdomain tool

Usage

root@Test2:~/tools/subfinder#cat subfinder.sh#!/bin/bashmkdir $1touch $1/$1.txtsubfinder -d $1 |tee /root/tools/subfinder/$1/$1.txtroot@Test2:~/tools/subfinder#./subfinder.sh twitch.tv

Amass & Subfinder are enough

Their output is comparable. So script them together on the CLI, then sort and unique the output

Used together, they cover about 30 sources

Amass can run a little bit longer than subfinder if you include the permutation scanning but not too long to impact your workflow

Other tools Jason used to use but not anymore:

Enumall / Recon-NG (not great on sources or speed)

Aquatone (not great on sources) but aquatone-scan is useful

Sublist3r (same as above)

Anything else for scraping

ClouDflare Enum (although sometimes he “thinks about it”)

He doesn’t use it but finds it interesting because he doesn’t understand the black magic behind how it works

Not sure if it uses sources better than Amass & Subfinder but he doesn’t think so

Subdomain Brute Forcing

Old tools:

Fierce

Blacksheepwall

Dns-parellel-prober

Knock.py

They’ve been supplanted by Massdns

Can run a million line dictionary in 30 sec

Because it’s written in C and breaks up your wordlist into small pieaces & assigns each piece to a different DNS resolver in Parallel

Subfinder also uses multiple DNS resolvers

Might be as good as Massdns but Jason hasn’t tried it yet for bruteforcing

CommonSpeak

By @naffy & @shubs

Content discovery wordlists built with BigQuery

Subdomain data is awesome, Jason plans on adding it to all.txt

But the URL data (URL paths) for content discovery has been less useful. It’s very app specific

So Jason uses CommonSpeak for subdomain data but not URL data

He also uses scans.io

Jason’s setup

Use Massdns

time ./subbrute.py /root/work/bin/all.txt $TARGET.com | ./bin/massdns -r resolvers.txt -t A -a -o -w massdns_output.txt -

Try Subfinder

Use Gobuster as the best non multi-resolver tool

time gobuster -m dns -u $TARGET.com -t 100 -w all.txt

Environment setup:

DigitalOcean

$10/mo offer (1 GB RAM, 1 CPU, 30 GB SSD, 2TB transfer)

Ubuntu 16.04 LTS

Subdomains wordlist to use: all.txt

Also use CommonSpeak & scans.io for subdomain data

Other methods of finding subdomains

Dnssec / Nsec / Nsec3 walking

Ldnsutils, Nsec3walker, Nsec3map

Github recon

Search for goodies

Google Dorking

Ads key, private policy, TOS, AWS, S3

See the “Esoteric sub-domain enumeration techniques” & “Doing recon like a boss” talks

Enumerating targets

Port Scanning

Use Masscan

By Robert Graham

Fastest port scanner. Nmap takes forever if you do a full port scan on all hosts of an ASN

Usage: masscan -p1-65535 -iL $TARGET_LIST --max-rate 10000 -oG $TARGET_OUTPUT

Shell script to run dig on a domain, strip out the HTTP/HTTPS prefix then run Masscan

Because Masscan takes only IPs as input, not DNS names

Use it to run Masscan against either a name domain or an IP range

#!/bin/bashstrip=$(echo$1|sed's/https\?:\/\///')echo ""echo "##################################################"host $stripecho "##################################################"echo ""masscan -p1-65535 $(dig +short$strip|grep-oE"\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"|head-1)--max-rate 1000 |& tee $strip_scan

Credential bruteforce

Masscan -> Nmap service scan-og -> Brutespray credential bruteforce

Use Masscan with the -oG option to get an output in Nmap format

Re-scan the output with Nmap version scanning

Pass the output to Brutespray which will bruteforce any remote administration protocol found with default & common passwords, and anonymous logins

Usage: python brutespray.py --file nmap.gnmap -U /usr/share/wordlist/user.txt -P /usr/share/wordlist/pass.txt --threads 5 --hosts 5

Visual identification

Not all subdomains previously scraped off the internet are still up, some are the same (DNS redirects) & we don’t know which protocol they are on (HTTP or HTTPS)

We could port scan ports 80 & 443

But another way to find out what is up and unique is taking screenshots with Eyewitness

Eyewitness

Eyewitness takes a list of domains without a protocol, visits each one with a headless browser, takes a screenshot & dumps them to a report

Usage: python Eyewitness.py --prepend-https -f ../domain/tesla.com.lst --all-protocols --headless

=> tells you which domains redirect to the same app, which domains are interesting & should be hacked first

Why not Aquatone or Httpscreenshot: Because Eyewitness tries both HTTP & HTTPS protocols

Eyewitness isn’t foolproof: The headless browser doesn’t always resolve, timeout issues, it can take a while…

=> Use Eyewitness for large targets.

=> But if you have a small list (~20 hosts), use sed or awk to add HTTP & HTTPS to every subdomain, then load them up manually with the CLI in the browser

Or use OpenList, a Chrome extension: Give it your list of URLs & it’ll open them in multiple tabs

What to look for?

Not interesting:

Anything that redirects to the main site

Help portal, email portal or OWA portal. They’re usually hardened out of the box (not always but most of them time)

Interesting:

Custom apps that are not the main website

They’ve probably spent less time with security on those sites than the main domain

Non standard Web ports:

There’s a list of ~20 very common non-standard Web ports (i.e. services not running on port 80 or 443)

You can find them by port scanning with Masscan & service scanning with Nmap

Then add them to Eyewitness with the --add-http-ports or --add-https-ports options.

Wayback enumeration

Also useful when you find subdomains that look juicy but don’t resolve

Example: Site protected with Basic Auth but an old version cached by Wayback Machine has configuration information of a server on the front page.

Tools

ReconCat by Dawood Ikhlaq

[Waybackurls by Tom Hudson

The one Jason uses the most, for pulling one domain from archive.org’s history

WaybackUnifier by Mohammed Diaa‏

Platform Identification & CVE searching

Retire.js: Outdated libraries (cmd-line, Burp on online form)

Builtwith: Stack information profiling

Wappalyzer: Similar to Builtwith (cmd-line, browser extension or online form)

burp-vulners-scanner: Burp plugin, detects versions with CVEs

A lot of memory is needed to use many Burp extensions on large scope bounties !

Parsing JavaScript

Generally automation doesn’t handle JavaScript very well

You could parse JS files manually but it’s not possible on large scope bounties

Many people assume Burp automatically parses JS files, relative paths, etc, and is able to execute all JS it finds. Burp is good but not perfect for this

Tools for better coverage of heavy JS sites:

ZAP Ajax Spider

Basically spiders the site with a headless browser

LinkFinder

Extracts absolute & relative URLs from JS files

JSParser

Burp pro only

Conference notes: The Bug Hunters Methodology v3(ish) (LevelUp 0x02 / 2018)

你可能感兴趣的:(Conference notes: The Bug Hunters Methodology v3(ish) (LevelUp 0x02 / 2018))