zgrab2 HTTP 模块二次开发

zgrab2是非常优秀的开源项目，可以对很多协议进行抓取，包括HTTP，但使用过程中遇到的小问题，如下记录下来

zgrab2 获得原始HTTP返回头

查看zgrab2的返回结果,会将HTTP返回头的key全部进行小写,同时将-修改为- ，比如将X-Powered-By 修改为 x_powered_by,这对于后期需要根据返回头进行判断的场景下，存在干扰
具体实现在

zgrab2-master\lib\http\header.go

func FormatHeaderName(s string) string {
	return strings.Replace(strings.ToLower(s), "-", "_", 30)
}

需要将

return strings.Replace(strings.ToLower(s), "-", "_", 30)

修改为

return s 

这样将返回头信息,保存到 Unknown 结果中，如下返回结果所示:

{"domain":"ip-api.com/json/111.47.226.119","data":{"http":{"status":"success","protocol":"http","result":{"response":{"status_line":"200 OK","status_code":200,"protocol":{"name":"HTTP/1.1","major":1,"minor":1},"headers":{"Unknown":[{"key":"Access-Control-Allow-Origin","value":["*"]},{"key":"X-Ttl","value":["60"]},{"key":"X-Rl","value":["44"]},{"key":"Date","value":["Mon, 13 Dec 2021 10:52:10 GMT"]},{"key":"Content-Type","value":["application/json; charset=utf-8"]},{"key":"Content-Length","value":["69"]}]},"body":"{\"status\":\"fail\",\"message\":\"invalid query\",\"query\":\"111.47.226.119/\"}","body_sha256":"3b5955d6d94291bc7dc1b3c6e95c0e2a945644f8b12e5946247839bf4d6d0cf0","content_length":69,"request":{"url":{"scheme":"http","host":"ip-api.com","path":"/json/111.47.226.119/"},"method":"GET","headers":{"Unknown":[{"key":"Accept","value":["*/*"]},{"key":"User-Agent","value":["Mozilla/5.0 zgrab/0.x"]}]},"host":"ip-api.com"}}},"timestamp":"2021-12-13T18:52:10+08:00"}}}

具体代码在

zgrab2-master\lib\http\header.go  
func filterHeaders(h Header) {
	var unknownHeaders []UnknownHeader
	for header, values := range h {
		if _, ok := knownHeaders[FormatHeaderName(header)]; !ok { //判断是否已知返回头
			unk := UnknownHeader{
				Key:    FormatHeaderName(header), 
				Values: values,
			}
			unknownHeaders = append(unknownHeaders, unk) //记录到未识别返回头中
			h.Del(header)  // 从返回头中进行删除
		}
	}
	if len(unknownHeaders) > 0 {
		if unknownHeaderStr, err := json.Marshal(unknownHeaders); err == nil {
			h["Unknown"] = []string{string(unknownHeaderStr)}
		}
	}
}

该函数主要判断返回头key小写值是否在内置的数组中，如果不在就放到Unknown新key中 ,目的主要是美化返回头值, 这个函数意义不是特别大

如果不想将返回头 保存到 Unknow这个 字段中,则继续看 filterHeaders 函数的调用

zgrab2-master\lib\http\header.go 
// Custom JSON Marshaller to comply with snake_case header names
func (h Header) MarshalJSON() ([]byte, error) {
	filterHeaders(h)

	headerMap := make(map[string]interface{})
	for k, v := range h {
		// Need to special-case unknown header object, since it's not a true header (aka map[string][]string)
		if k == "Unknown" && len(v) > 0 {
			var unknownHeader []UnknownHeader
			json.Unmarshal([]byte(v[0]), &unknownHeader)
			for idx := range unknownHeader {
				formatHeaderValues(unknownHeader[idx].Values)
			}
			headerMap[FormatHeaderName(k)] = unknownHeader
		} else {
			formatHeaderValues(v)
			headerMap[FormatHeaderName(k)] = v
		}
	}

	return json.Marshal(headerMap)
}

由于filterHeaders 没有特别大意义，所以直接将 filterHeaders(h) 注释掉, 最终获得HTTP返回header的 代码 headerMap[FormatHeaderName(k)] = v ,所以保留原始HTTP返回头的方法
使用新的 MarshalJSON 即可

zgrab2-master\lib\http\header.go 
// Custom JSON Marshaller to comply with snake_case header names
func (h Header) MarshalJSON() ([]byte, error) {
//	filterHeaders(h)

	headerMap := make(map[string]interface{})
	for k, v := range h {
		// Need to special-case unknown header object, since it's not a true header (aka map[string][]string)
		if k == "Unknown" && len(v) > 0 {
			var unknownHeader []UnknownHeader
			json.Unmarshal([]byte(v[0]), &unknownHeader)
			for idx := range unknownHeader {
				formatHeaderValues(unknownHeader[idx].Values)
			}
			headerMap[FormatHeaderName(k)] = unknownHeader
		} else {
			formatHeaderValues(v)
			// headerMap[FormatHeaderName(k)] = v
			headerMap[(k)] = v
		}
	}

	return json.Marshal(headerMap)
}

最终返回结果类似如下:

{"domain":"ip-api.com/json/111.47.226.119","data":{"http":{"status":"success","protocol":"http","result":{"response":{"status_line":"200 OK","status_code":200,"protocol":{"name":"HTTP/1.1","major":1,"minor":1},"headers":{"Access-Control-Allow-Origin":["*"],"Content-Length":["69"],"Content-Type":["application/json; charset=utf-8"],"Date":["Mon, 13 Dec 2021 14:03:46 GMT"],"X-Rl":["44"],"X-Ttl":["60"]},"body":"{\"status\":\"fail\",\"message\":\"invalid query\",\"query\":\"111.47.226.119/\"}","body_sha256":"3b5955d6d94291bc7dc1b3c6e95c0e2a945644f8b12e5946247839bf4d6d0cf0","content_length":69,"request":{"url":{"scheme":"http","host":"ip-api.com","path":"/json/111.47.226.119/"},"method":"GET","headers":{"Accept":["*/*"],"User-Agent":["Mozilla/5.0 zgrab/0.x"]},"host":"ip-api.com"}}},"timestamp":"2021-12-13T22:03:45+08:00"}}}

需要注意:headers 每个key对应值是一个列表,有些key可能有多个返回值，将会进行合并

删除 zgrab2 结果中的request字段

为了将返回结果尽可能的小,同时request是客户端配置，理论上都知道内容，不需要记录到返回结果中,可以将返回结果的request字段进行删除
具体操作方法:

zgrab2-master\lib\http\response.go

func ReadResponse(r *bufio.Reader, req *Request) (*Response, error) {
	tp := textproto.NewReader(r)
	resp := &Response{
		Request: req,    //将这行注释掉即可 
	}

其他Response字段下的返回结果，都可以在该函数进行挑战