本文主要介绍下在腾讯云上搭建squid代理服务器,用于访问国外网站或者为爬虫提供代理ip,以及简单介绍下如何基于腾讯云提供的SDK,批量开启或者销毁代理服务器实例。
Squid是一个高性能的代理缓存服务器,Squid支持FTP、gopher、HTTPS和HTTP协议。和一般的代理缓存软件不同,Squid用一个单独的、非模块化的、I/O驱动的进程来处理所有的客户端请求。
下面是搭建步骤:
1、yum安装软件,并设置squid开机自启
yum install -y squid yum install -y httpd systemctl enable squid.service
2、创建squid代理的访问用户,并设置好密码
htpasswd -c /etc/squid/passwd 用户名
需要输入两次密码
3、配置squid.conf,并重启代理服务器
# # Recommended minimum configuration: # # Example rule allowing access from your local networks. # Adapt to list your (internal) IP networks from where browsing # should be allowed acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT # # Recommended minimum Access Permission configuration: # # Deny requests to certain unsafe ports http_access deny !Safe_ports # Deny CONNECT to other than secure SSL ports http_access deny CONNECT !SSL_ports # Only allow cachemgr access from localhost http_access allow localhost manager http_access deny manager # We strongly recommend the following be uncommented to protect innocent # web applications running on the proxy server who think the only # one who can access services on "localhost" is a local user #http_access deny to_localhost # # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS # # Example rule allowing access from your local networks. # Adapt localnet in the ACL section to list your (internal) IP networks # from where browsing should be allowed http_access allow localnet http_access allow localhost # And finally deny all other access to this proxy #http_access deny all # Squid normally listens to port 3128 http_port 808 # Uncomment and adjust the following to add a disk cache directory. #cache_dir ufs /var/spool/squid 100 16 256 # Leave coredumps in the first cache dir coredump_dir /var/spool/squid # # Add any of your own refresh_pattern entries above these. # refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache_mem 128 MB maximum_object_size 16 MB cache_dir ufs /var/spool/squid 100 16 256 access_log /var/log/squid/access.log auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid/passwd auth_param basic children 5 auth_param basic realm Squid auth_param basic credentialsttl 5 hours acl ncsa_users proxy_auth REQUIRED http_access allow ncsa_users http_access deny all visible_hostname Squid.org cache_mgr abingtech@163.com
systemctl restart squid.service
4、从上面的配置文件可以看出,访问端口为808,需要在云服务器的安全组中开放端口
5、可以在浏览器设置中配置代理服务器或者使用代理工具进行测试验证
为了管理方便,采用基于腾讯云SDK进行开发,我使用的是Java语言,代码很简单就不进行过多介绍,有兴趣的同学可以从码云
【https://gitee.com/abingtech/proxy.git】上pull,这里主要说明下需要注意的点:
1、申请SecretId和SecretKey
2、搭建好代理服务器后,需要在腾讯云上手工制作好镜像,作为批量创建实例的模版
3、由于腾讯云设置的每次返回列表最大limit是100,这里需要自己手工处理分页的情况
另外,云服务器实例建议买按流量计费的模式,这样不用随时可以销毁,要用启动下就行了。