curl--使用详细记录
类似的一个开源项目：https://github.com/httpie/httpie
常见用法
1. curl http://myip.ipip.net
直接访问
2. curl -A "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90Safari/537.36" http://myip.ipip.net
-A 参数指定客户端用户代理的标识头，UA，默认是curl/7.68.0
3. curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90Safari/537.36" http://myip.ipip.net
-H 也可以直接HTTP请求头，更改User-Agent
4. curl -b 'cntf=fuck' https://myip.ipip.net
-b 用来向服务器发送cookie
5. curl -c cookies.txt https://myip.ipip.net
-c 将服务端设置的cookie在本地生成一个文件写入进去
6. curl -d 'login=bab&password=shit' -x POST https://myip.ipip.net
-d 参数用于发送 POST 请求的数据体(body体) 使用-d参数，HTTP 请求会自动加上标头Content-Type : application/x-www-form-urlencoded。并且会自动将请求转为 POST 方法，因此可以省略-X POST
7. curl -d '@cookies.txt' https://myip.ipip.net
-d 参数也可以读取本地文本数据，向服务端发送
8. curl --data-urlencode 'cntf=shit hehe' http://myip.ipip.net
--data-urlencode 参数等同于-d参数，也是发送POST请求数据，区别是会自动将需要发送的数据进行URL编码
9. curl -e 'https://www.qq.com' https://myip.ipip.net
-e 参数用来设置HTTP的请求头Referer用来表示请求来源，当然这里也可以使用-H参数来表示，区别是要把完整的请求头写出来，包括Referer写进去，而-e参数不需要写入Referer
10. curl -F 'file=@cntf' https://myip.ipip.net
-F 参数可以用来向服务器上传二进制文件，这里cntf就是要给二进制文件，上面的命令在HTTP请求中加上了头Content-Type: multipart/form-data 然后将文件cntf作为file字段上传
11. curl -F '[email protected];type=image/png' https://myip.ipip.net
-F 参数也可以指定MIME类型
12. curl -G -d 'wd=test' -d 'count=30' https://www.baidu.com/s
-G 参数用来构造URL查询字符串 类似构造成这个 https://www.baidu.com/s?wd=test&count=20
13. curl -i https://myip.ipip.net
-i 参数打印出服务器响应的HTTP头，上面的命令操作完成之后会先输出服务器响应的头，然后再空一行，再输出网页源码
14. curl -I https://www.qq.com
-I 参数向服务器发送HEAD请求，然后将服务器返回的HTTP响应头打印出来，不返回输出源码信息
15. curl --head https://www.qq.com
--head 参数跟-I参数一样的功能
16. curl -k https://myip.ipip.net
-k 参数指定跳过SSL检测，上面的命令不会检查服务器的SSL证书是否正确
17. curl -L http://qq.com
-L 参数会让HTTP请求跟随服务器重定向，默认情况下curl是不会跟随重定向的
18. curl --limit-rate 1k https://www.qq.com
--limit-rate 参数是用来限制HTTP请求和响应的带宽大小，一把用来模拟网速慢得环境，上面得命令表示将带宽限制在每秒1k字节
19. curl -o test.html https://myip.ipip.net
-o 参数是将服务器的响应内容保存成文件，相当于wget命令来下载文件了，上面的命令是将响应的内容保存成test.html文件
20. curl -O https://www.ruanyifeng.com/blog/2019/09/curl-reference.html
-O 参数将服务器响应内容保存成文件，并URL的最后一部分当作文件名称
21. curl -s https://myip.ipip.net1
-s 参数是将不输出错误和进度信息，上面的命令是故意写错域名，访问结果是不返回任何信息，如果不发送错误，会显示正确的结果
22. curl -s -o /dev/null https://myip.ipip.net
-s&-o /dev/null 参数加了/dev/null表示不管正确错误与否都不显示出来
23. curl -u 'cntf:cntf' https://myip.ipip.net
-u 参数用来设置服务器认证的用户名和密码，上面的命令操作完成之后会将HTTP头变成401认证那种形式类似Authorization: Basic xxxxxx
其他形式：
curl https://cntf:cntf@https://myip.ipip.net
或者
curl -u 'cntf' https://myip.ipip.net 完成之后会让你输入密码
24. curl -v https://myip.ipip.net
-v 参数用于输出整个通信过程，一把用于调试
25. curl --trace - https://myip.ipip.net
--trace - 参数也是可以用于调试，而且还会显示输出原始的二进制数据内容
26. curl -x socks5://10.0.1.251:10808 https://www.facebook.com
-x 参数只当http请求代理，上面的命令是使用socks5代理，而且没有用户名和密码认证
27. curl -x socks5://cntf:[email protected]:10808 https://www.facebook.com
上面是带认证的socks5代理
28. curl -x 10.0.1.251:10808 https://www.facebook.com
如果没有指定使用哪种代理协议，默认情况下是http代理协议
29. curl -X POST https://myip.ipip.net
-X 参数用于指定HTTP请求的请求方法
30. curl --trace output.txt https://myip.ipip.net
--trace 参数调试过程输出到文本中，里面有16进制内容，正常显示响应内容
31. curl --trace-ascii out.txt https://myip.ipip.net
--trace-ascii 跟上面结果一样，区别是没有16进制内容
32. curl -X POST --data-urlencode "date=May 13" https://myip.ipip.net
-X POST --data-urlencode POST请求进行url编码
33. curl --referer https://shit.qq.com https://myip.ipip.net
--referer 在HTTP请求中添加referer头
34. curl --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90Safari/537.36" https://myip.ipip.net
--user-agent 参数同样是指定user-agent头
35. curl --header "Content-Type:application/json" https://myip.ipip.net
--header 参数指定一个HTTP请求头
36. curl -d 'login=cntf' -d 'password=cntfshit' https://myip.ipip.net
-d 使用多个-d参数配置更整洁的POST请求，使用了-d参数就可以省略添加POST的操作
37. curl -d '{"login": "cntf", "pass":"cntfshit"}' -H 'Content-Type: application/json' https://myip.ipip.net
-d 使用POST请求提交一个json格式的数据
38. curl -d 'cntfcntfshit' -H 'Content-Type: text/xml' https://myip.ipip.net
-d 使用POST请求提交一个xml格式的数据
39. curl -d 'hello cntf' -H 'Content-Type: text/plain' https://myip.ipip.net
-d 使用POST请求提交一个纯文本内容
40. curl -d '@data.txt' https://myip.ipip.net
-d 使用POST请求从一个文本文件加载内容并提交
41. curl --data-urlencode 'comment=hello cntf' https://myip.ipip.net
--data-urlencode 使用POST请求提交一个url编码的信息
42. curl -A '' https://myip.ipip.net
-A 移除一个user-agent
43. curl -A '' -H 'User-Agent:' https://myip.ipip.net
-A -H 发送一个空User-Agent
44. curl -c cookies.txt https://myip.ipip.net
-c 参数保存cookie内容到一个文件
45. curl -b cookies.txt https://myip.ipip.net
-b 参数从一个文件内容加载cookies
46. curl -e 'https://shit.hehe.com' https://myip.ipip.net
-e 参数添加一个referer头
47. curl -e '' https://myip.ipip.net
-e 发送一个空referer头
48. curl -H 'https://shit.hehe.com' https://myip.ipip.net
-H 参数-H添加referer头
49. curl -s -o /dev/null -D - https://myip.ipip.net
-s -o /dev/null -D - 参数只输出响应头信息
50. curl -x socks4:10.0.1.251:10808 https://www.google.com
-x socks4 参数使用socks4代理无需认证
51. curl -1 https://myip.ipip.net
-1 参数数字-1表示使用SSLv1(TLSv1) ，数字-0是使用HTTP1.0 数字-2表示curl使用SSLv2(SSL) -3 表示curl使用SSLv3(SSL)
52. curl --trace - --trace-time https://myip.ipip.net
--trace - --trace-time 参数显示详细的访问开始时间和结束时间
53. curl -v -s -o /dev/null --stderr - https://myip.ipip.net | grep '^>'
-v -s -o /dev/null --stderr - 仅显示HTTP请求头信息
54. curl -w '%{response_code}' -s -o /dev/null https://myip.ipip..net
-w '%{response_code}' -s -o /dev/null 仅显示HTTP响应码
55. curl -# -o centos7.2009-everything.iso -C - http://mirrors.aliyun.com/centos/7/isos/x86_64/CentOS-7-x86_64-Everything-2009.iso
-o 显示下载进度,断点续传
56. curl -# -o centos7.2009-everything.iso http://mirrors.aliyun.com/centos/7/isos/x86_64/CentOS-7-x86_64-Everything-2009.iso
-o 简单进度条显示
参考：
https://catonmat.net/cookbooks/curl
man手册
https://curl.se/docs/manpage.html
kali@kali:~$ curl --help
Usage: curl [options...]
--abstract-unix-socket Connect via abstract Unix domain socket
--alt-svc Enable alt-svc with this cache file
--anyauth Pick any authentication method
-a, --append Append to target file when uploading
--basic Use HTTP Basic Authentication
--cacert CA certificate to verify peer against
--capath CA directory to verify peer against
-E, --cert Client certificate file and password
--cert-status Verify the status of the server certificate
--cert-type Certificate file type (DER/PEM/ENG)
--ciphers SSL ciphers to use
--compressed Request compressed response
--compressed-ssh Enable SSH compression
-K, --config Read config from a file
--connect-timeout Maximum time allowed for connection
--connect-to Connect to host
-C, --continue-at Resumed transfer offset
-b, --cookie Send cookies from string/file
-c, --cookie-jar Write cookies to after operation
--create-dirs Create necessary local directory hierarchy
--crlf Convert LF to CRLF in upload
--crlfile Get a CRL list in PEM format from the given file
-d, --data HTTP POST data
--data-ascii HTTP POST ASCII data
--data-binary HTTP POST binary data
--data-raw HTTP POST data, '@' allowed
--data-urlencode HTTP POST data url encoded
--delegation GSS-API delegation permission
--digest Use HTTP Digest Authentication
-q, --disable Disable .curlrc
--disable-eprt Inhibit using EPRT or LPRT
--disable-epsv Inhibit using EPSV
--disallow-username-in-url Disallow username in url
--dns-interface Interface to use for DNS requests
--dns-ipv4-addr IPv4 address to use for DNS requests
--dns-ipv6-addr IPv6 address to use for DNS requests
--dns-servers DNS server addrs to use
--doh-url Resolve host names over DOH
-D, --dump-header Write the received headers to
--egd-file EGD socket path for random data
--engine Crypto engine to use
--etag-save Get an ETag from response header and save it to a FILE
--etag-compare Get an ETag from a file and send a conditional request
--expect100-timeout How long to wait for 100-continue
-f, --fail Fail silently (no output at all) on HTTP errors
--fail-early Fail on first transfer error, do not continue
--false-start Enable TLS False Start
-F, --form Specify multipart MIME data
--form-string Specify multipart MIME data
--ftp-account Account data string
--ftp-alternative-to-user String to replace USER [name]
--ftp-create-dirs Create the remote dirs if not present
--ftp-method Control CWD usage
--ftp-pasv Use PASV/EPSV instead of PORT
-P, --ftp-port Use PORT instead of PASV
--ftp-pret Send PRET before PASV
--ftp-skip-pasv-ip Skip the IP address for PASV
--ftp-ssl-ccc Send CCC after authenticating
--ftp-ssl-ccc-mode Set CCC mode
--ftp-ssl-control Require SSL/TLS for FTP login, clear for transfer
-G, --get Put the post data in the URL and use GET
-g, --globoff Disable URL sequences and ranges using {} and []
--happy-eyeballs-timeout-ms How long to wait in milliseconds for IPv6 before trying IPv4
--haproxy-protocol Send HAProxy PROXY protocol v1 header
-I, --head Show document info only
-H, --header Pass custom header(s) to server
-h, --help This help text
--hostpubmd5 Acceptable MD5 hash of the host public key
--http0.9 Allow HTTP 0.9 responses
-0, --http1.0 Use HTTP 1.0
--http1.1 Use HTTP 1.1
--http2 Use HTTP 2
--http2-prior-knowledge Use HTTP 2 without HTTP/1.1 Upgrade
--http3 Use HTTP v3
--ignore-content-length Ignore the size of the remote resource
-i, --include Include protocol response headers in the output
-k, --insecure Allow insecure server connections when using SSL
--interface Use network INTERFACE (or address)
-4, --ipv4 Resolve names to IPv4 addresses
-6, --ipv6 Resolve names to IPv6 addresses
-j, --junk-session-cookies Ignore session cookies read from file
--keepalive-time Interval time for keepalive probes
--key Private key file name
--key-type Private key file type (DER/PEM/ENG)
--krb Enable Kerberos with security
--libcurl Dump libcurl equivalent code of this command line
--limit-rate Limit transfer speed to RATE
-l, --list-only List only mode
--local-port Force use of RANGE for local port numbers
-L, --location Follow redirects
--location-trusted Like --location, and send auth to other hosts
--login-options Server login options
--mail-auth Originator address of the original email
--mail-from Mail from this address
--mail-rcpt Mail to this address
-M, --manual Display the full manual
--max-filesize Maximum file size to download
--max-redirs Maximum number of redirects allowed
-m, --max-time Maximum time allowed for the transfer
--metalink Process given URLs as metalink XML file
--negotiate Use HTTP Negotiate (SPNEGO) authentication
-n, --netrc Must read .netrc for user name and password
--netrc-file Specify FILE for netrc
--netrc-optional Use either .netrc or URL
-:, --next Make next URL use its separate set of options
--no-alpn Disable the ALPN TLS extension
-N, --no-buffer Disable buffering of the output stream
--no-keepalive Disable TCP keepalive on the connection
--no-npn Disable the NPN TLS extension
--no-progress-meter Do not show the progress meter
--no-sessionid Disable SSL session-ID reusing
--noproxy List of hosts which do not use proxy
--ntlm Use HTTP NTLM authentication
--ntlm-wb Use HTTP NTLM authentication with winbind
--oauth2-bearer OAuth 2 Bearer Token
-o, --output Write to file instead of stdout
-Z, --parallel Perform transfers in parallel
--parallel-immediate Do not wait for multiplexing (with --parallel)
--parallel-max Maximum concurrency for parallel transfers
--pass Pass phrase for the private key
--path-as-is Do not squash .. sequences in URL path
--pinnedpubkey FILE/HASHES Public key to verify peer against
--post301 Do not switch to GET after following a 301
--post302 Do not switch to GET after following a 302
--post303 Do not switch to GET after following a 303
--preproxy [protocol://]host[:port] Use this proxy first
-#, --progress-bar Display transfer progress as a bar
--proto Enable/disable PROTOCOLS
--proto-default Use PROTOCOL for any URL missing a scheme
--proto-redir Enable/disable PROTOCOLS on redirect
-x, --proxy [protocol://]host[:port] Use this proxy
--proxy-anyauth Pick any proxy authentication method
--proxy-basic Use Basic authentication on the proxy
--proxy-cacert CA certificate to verify peer against for proxy
--proxy-capath CA directory to verify peer against for proxy
--proxy-cert Set client certificate for proxy
--proxy-cert-type Client certificate type for HTTPS proxy
--proxy-ciphers SSL ciphers to use for proxy
--proxy-crlfile Set a CRL list for proxy
--proxy-digest Use Digest authentication on the proxy
--proxy-header Pass custom header(s) to proxy
--proxy-insecure Do HTTPS proxy connections without verifying the proxy
--proxy-key Private key for HTTPS proxy
--proxy-key-type Private key file type for proxy
--proxy-negotiate Use HTTP Negotiate (SPNEGO) authentication on the proxy
--proxy-ntlm Use NTLM authentication on the proxy
--proxy-pass Pass phrase for the private key for HTTPS proxy
--proxy-pinnedpubkey FILE/HASHES public key to verify proxy with
--proxy-service-name SPNEGO proxy service name
--proxy-ssl-allow-beast Allow security flaw for interop for HTTPS proxy
--proxy-tls13-ciphers TLS 1.3 ciphersuites for proxy (OpenSSL)
--proxy-tlsauthtype TLS authentication type for HTTPS proxy
--proxy-tlspassword TLS password for HTTPS proxy
--proxy-tlsuser TLS username for HTTPS proxy
--proxy-tlsv1 Use TLSv1 for HTTPS proxy
-U, --proxy-user Proxy user and password
--proxy1.0 Use HTTP/1.0 proxy on given port
-p, --proxytunnel Operate through an HTTP proxy tunnel (using CONNECT)
--pubkey SSH Public key file name
-Q, --quote Send command(s) to server before transfer
--random-file File for reading random data from
-r, --range Retrieve only the bytes within RANGE
--raw Do HTTP "raw"; no transfer decoding
-e, --referer Referrer URL
-J, --remote-header-name Use the header-provided filename
-O, --remote-name Write output to a file named as the remote file
--remote-name-all Use the remote file name for all URLs
-R, --remote-time Set the remote file's time on the local output
-X, --request Specify request command to use
--request-target Specify the target for this request
--resolve Resolve the host+port to this address
--retry Retry request if transient problems occur
--retry-connrefused Retry on connection refused (use with --retry)
--retry-delay Wait time between retries
--retry-max-time Retry only within this period
--sasl-authzid Use this identity to act as during SASL PLAIN authentication
--sasl-ir Enable initial response in SASL authentication
--service-name SPNEGO service name
-S, --show-error Show error even when -s is used
-s, --silent Silent mode
--socks4 SOCKS4 proxy on given host + port
--socks4a SOCKS4a proxy on given host + port
--socks5 SOCKS5 proxy on given host + port
--socks5-basic Enable username/password auth for SOCKS5 proxies
--socks5-gssapi Enable GSS-API auth for SOCKS5 proxies
--socks5-gssapi-nec Compatibility with NEC SOCKS5 server
--socks5-gssapi-service SOCKS5 proxy service name for GSS-API
--socks5-hostname SOCKS5 proxy, pass host name to proxy
-Y, --speed-limit Stop transfers slower than this
-y, --speed-time Trigger 'speed-limit' abort after this time
--ssl Try SSL/TLS
--ssl-allow-beast Allow security flaw to improve interop
--ssl-no-revoke Disable cert revocation checks (Schannel)
--ssl-reqd Require SSL/TLS
-2, --sslv2 Use SSLv2
-3, --sslv3 Use SSLv3
--stderr Where to redirect stderr
--styled-output Enable styled output for HTTP headers
--suppress-connect-headers Suppress proxy CONNECT response headers
--tcp-fastopen Use TCP Fast Open
--tcp-nodelay Use the TCP_NODELAY option
-t, --telnet-option Set telnet option
--tftp-blksize Set TFTP BLKSIZE option
--tftp-no-options Do not send any TFTP options
-z, --time-cond
1. aggregateByKey的运行机制
/**
* Aggregate the values of each key, using given combine functions and a neutral "zero value".
* This function can return a different result type
spark-sql是Spark bin目录下的一个可执行脚本，它的目的是通过这个脚本执行Hive的命令，即原来通过
hive>输入的指令可以通过spark-sql>输入的指令来完成。
spark-sql可以使用内置的Hive metadata-store，也可以使用已经独立安装的Hive的metadata store
关于Hive build into Spark
// Max value in Array
var arr = [1,2,3,5,3,2];Math.max.apply(null, arr); // 5
// Max value in Jaon Array
var arr = [{"x":"8/11/2009","y":0.026572007},{"x"
在使用XMlhttpRequest对象发送请求和响应之前，必须首先使用javaScript对象创建一个XMLHttpRquest对象。
var xmlhttp；
function getXMLHttpRequest(){
if(window.ActiveXObject){
xmlhttp:new ActiveXObject("Microsoft.XMLHTTP
