三、简单文件服务器，自动异步缩略图
1.log.conf，python的日志模块，详见：http://www.red-dove.com/python_logging.html
[loggers]
keys=root,upload,resize
[handlers]
keys=consoleHandler,uploadFileHandler,resizeFileHandler
[formatters]
keys=simpleFormatter
[formatter_simpleFormatter]
format=%(asctime)s %(levelname)s %(message)s
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_upload]
level=DEBUG
handlers=uploadFileHandler
qualname=upload
[logger_resize]
level=DEBUG
handlers=resizeFileHandler
qualname=resize
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)
[handler_uploadFileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('upload.log',"midnight", 1)
[handler_resizeFileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('resize.log',"midnight", 1)
跟java中的log4j很类似，定义三个logger对象：root,upload,resize，其中root是StreamHandler类型，使用sys.out；upload和resize是TimedRotatingFileHandler，每天生成一个日志文件的方式。
2.conf.py配置文件
FILE_BIG_PATH = '/data/hiluofilev2/big/' FILE_SMALL_PATH = '/data/hiluofilev2/small/' HEAD_BIG_PATH = '/data/hiluohead/big/' HEAD_SMALL_PATH = '/data/hiluohead/small/' GEARMAN_SERVER = '127.0.0.1:4730' RESIZE_WORKER = 'resize' FILE_RESIZE = '140x160!' HEAD_RESIZE = '86x86!' IMAGE_FILE = ['jpg','png','gif','jpeg'] AUDIO_FILE = ['amr'] HTTP_FILE = 'http://192.168.0.181:9999/file/' HTTP_HEAD = 'http://192.168.0.181:9999/head/' import logging import logging.config import os logging.config.fileConfig(os.path.join(os.path.dirname(os.path.abspath(__file__)), "log.conf"))
FILE_BIG_PATH：这几个PATH表示原图和缩略图保存路径，根据业务区分，用户头像和用户上传的文件分开存放。
GEARMAN_SERVER：gearman Server端的地址。
RESIZE_WORKER：执行图片缩放的worker的名字。
FILE_RESIZE和HEAD_RESIZE：普通图片和头像的缩放尺寸。
IMAGE_FILE和AUDIO_FILE：系统支持的图片格式和音频文件格式，业务上仅支持两种类型。
HTTP_FILE和HTTP_HEAD：文件下载的HTTP前缀。
然后是初始化logging的功能。
3.worker_resize.py 生成缩略图的worker
#!/usr/bin/env python
import os
import gearman
import math
import conf
import logging
import pickle
from pgmagick import Image, FilterTypes
log_resize = logging.getLogger("resize")
class CustomGearmanWorker(gearman.GearmanWorker):
def on_job_execute(self, current_job):
log_resize.info("Resize Worker started")
return super(CustomGearmanWorker, self).on_job_execute(current_job)
def resize_task_callback(gearman_worker, job):
filein = job.data.split('-')[0]
fileout = job.data.split('-')[1]
fileresize = job.data.split('-')[2]
log_resize.info('start filein:%s -> fileout:%s' % (filein,fileout))
im = Image(filein)
im.quality(80)
im.filterType(FilterTypes.SincFilter)
im.scale(fileresize)
im.sharpen(1.0)
im.write(fileout)
log_resize.info('end filein:%s -> fileout:%s' % (filein,fileout))
return job.data
new_worker = CustomGearmanWorker([conf.GEARMAN_SERVER])
new_worker.register_task("resize", resize_task_callback)
new_worker.work()
jobdata是string，filein代表原图路径，fileout代表缩略图路径，fileresize代表缩放尺寸
4.upload.py 用于上传文件和测试上传文件，使用web.py实现
#!/usr/bin/env python
# coding: utf-8
import web
import conf
import logging
import gearman
urls = (
'/', 'Index',
'/upload', 'FileUpload'
)
app = web.application(urls, globals())
log_upload = logging.getLogger("upload")
# get some directories in conf file
filebigdir = conf.FILE_BIG_PATH
filesmalldir = conf.FILE_SMALL_PATH
headbigdir = conf.HEAD_BIG_PATH
headsmalldir = conf.HEAD_SMALL_PATH
resize_client = gearman.GearmanClient([conf.GEARMAN_SERVER])
class Index:
def GET(self):
web.header("Content-Type","text/html; charset=utf-8")
return """<html><head></head><body>this hiluofileV2</body></html>"""
class FileUpload:
def GET(self):
web.header("Content-Type","text/html; charset=utf-8")
return """<html><head></head><body>
<form method="POST" enctype="multipart/form-data" action="">
username:<input type="text" name="username" size="20" /><br/>
md5 or sha1:<input type="text" name="md5orsha1" size="80"/><br/>
<input type="file" name="myfile" />
<br/>
<input type="submit" />
</form>
</body></html>"""
def POST(self):
client = web.ctx.environ['REMOTE_ADDR']
x = web.input(myfile={},username="",md5orsha1="")
username = x.username.encode('utf-8')
intusername = 0
try:
intusername = int(username) # check if the username is right
except ValueError:
log_upload.error('ip:%s username:%s username not right!' % (client,username))
return 'error[username]'
md5orsha1 = x.md5orsha1.encode('utf-8') # check if the md5orsha1 is right
if len(md5orsha1)==32 or len(md5orsha1)==40:
if 'myfile' in x: # check if the file-object is created
filepath = x.myfile.filename.replace('\\','/')
filename = filepath.split('/')[-1]
filesuffix = filename.split('.')[-1].lower()
filename = md5orsha1.lower() + '.' + filesuffix # get saved filename(md5orsha1+‘.’+suffix)
filebig = filebigdir + filename
if filesuffix in conf.AUDIO_FILE: # audio file just save no need to resize
fout = open(filebig,'w') # creates the file where the uploaded file should be stored
fout.write(x.myfile.file.read()) # writes the uploaded file to the newly created file.
fout.close() # closes the file, upload complete.
log_upload.info('ip:%s username: %d upload: %s ok!' % (client,intusername,filename))
fileurl = conf.HTTP_FILE + 'big/' + filename # file download url
return fileurl
elif filesuffix in conf.IMAGE_FILE:
filesmall = filesmalldir + filename # thumbnail file
fileresize = conf.FILE_RESIZE # default FILE_RESIZE
fileurl = conf.HTTP_FILE + 'small/' + filename
if len(md5orsha1)==40: # user upload head
filebig = headbigdir + filename
filesmall = headsmalldir + filename # thumbnail head
fileresize = conf.HEAD_RESIZE # thumbnail head HEAD_RESIZE
fileurl = conf.HTTP_HEAD + 'small/' + filename
fout = open(filebig,'w') # creates the file where the uploaded file should be stored
fout.write(x.myfile.file.read()) # writes the uploaded file to the newly created file.
fout.close() # closes the file, upload complete.
log_upload.info('ip:%s username:%d upload:%s ok!' % (client,intusername,filename))
jobdata = ('%s-%s-%s' % (filebig,filesmall,fileresize))
resize_client.submit_job(conf.RESIZE_WORKER, jobdata) # submit_job resize and ignore job result
log_upload.info('submit resize:%s ok!' % (jobdata))
return fileurl
else:
log_upload.error('ip:%s username:%d upload:%s unsupport file type' % (client,intusername,filename))
return 'error[filetype]'
else:
log_upload.error('ip:%s username:%d upload no myfile' % (client,intusername))
return 'error[myfile]'
else:
log_upload.error('ip:%s username:%d upload:%s unsupport md5orsha1' % (client,intusername,md5orsha1))
return 'error[filemd5orsha1]'
if __name__ == "__main__":
#app = web.application(urls, globals())
web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)
app.run()
这里业务上定义传头像的key是40位的sha1编码，普通图像文件是32位的md5编码。
5.nginx的nginx.conf修改，使用fastCGI方式运行，
在http节点下增加：
proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; #获取真实IP client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k;
增加fastCGI静态图片的下载location，nginx开启8099端口代理fastCGI，9999端口用来下载文件
server{
listen 8099;
server_name fast;
charset utf-8;
location / {
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_pass 127.0.0.1:8091;
}
}
server
{
listen 9999;
server_name file.hiluo.cn;
location /file/ {
alias /data/hiluofilev2/;
}
location /head/ {
alias /data/hiluohead/;
}
}
注意，在配置nginx的fastcgi时可能会报错如下：
5.1.child exited with 2
解决方法: insert #!/usr/bin/env python into header of main.py
5.2.spawn-fcgi child exited with 126
解决方法: chmod +x upload.py
5.3.child exited with1
这时可以先修改upload.py，屏蔽web.wsgi.runwsgi行，打开app = web.application(urls, globals())行，用python upload.py 8091方式启动测试。
6.hiluo-file-http.sh，runwsgi的启动停止管理脚本
#!/bin/sh
#
# hiluo-file-http control
# $Date: 2012-07-04 willzhai $
#
# If pid file path is not set elsewhere, set to /tmp/hiluo-file-http.pid
[ -z "$PIDFILE" ] && PIDFILE="/tmp/hiluo-file-http.pid"
dir=/root/test-file-v2
start() {
echo "Starting hiluo-file-http..."
exec_result=`spawn-fcgi -d $dir -f $dir/upload.py -a 127.0.0.1 -p 8091`
echo $exec_result
if [ ! -z "$PIDFILE" ]; then
echo $exec_result|awk -F ':' '{print $4}' > $PIDFILE
fi
}
stop() {
# Stop daemons.
echo "Shutting down hiluo-file-http..."
kill `pgrep -f "python $dir/upload.py"`
rm -f $PIDFILE
}
restart() {
stop
sleep 3 # give it a few moments to shut down
start
}
status() {
pid=`cat $PIDFILE 2>&1`
if [ "$?" = "1" ]; then
echo "hiluo-file-http is not running"
RETVAL=0
else
ps -p $pid > /dev/null 2>&1
if [ "$?" = "0" ]; then
echo "hiluo-file-http is running"
RETVAL=0
else
echo "hiluo-file-http is not running"
RETVAL=0
fi
fi
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
status
;;
*)
echo "Usage $0 {start|stop|restart|status}"
RETVAL=1
esac
exit $?
注意将dir改成实际的部署路径。这个可以再改进，自动识别当前路径。
7.hiluo-file-resize.sh ，resize worker的启动停止管理脚本
#!/bin/sh
#
# hiluo-file-resize control
# $Date: 2012-07-04 willzhai $
#
# If pid file path is not set elsewhere, set to /tmp/hiluo-file-resize.pid
[ -z "$PIDFILE" ] && PIDFILE="/tmp/hiluo-file-resize.pid"
dir=/root/test-file-v2
start() {
echo "Starting hiluo-file-resize..."
exec_command="exec python $dir/worker_resize.py 2>&1 &"
eval $exec_command
RETVAL=$?
if [ $RETVAL -eq 0 -a ! -z "$PIDFILE" ]; then
echo $! > $PIDFILE
fi
echo "hiluo-file-resize new pid: "`cat $PIDFILE`
}
stop() {
# Stop daemons.
echo "Shutting down hiluo-file-resize..."
kill `pgrep -f "python $dir/worker_resize.py"`
rm -f $PIDFILE
}
restart() {
stop
sleep 3 # give it a few moments to shut down
start
}
status() {
pid=`cat $PIDFILE 2>&1`
if [ "$?" = "1" ]; then
echo "hiluo-file-resize is not running"
RETVAL=0
else
ps -p $pid > /dev/null 2>&1
if [ "$?" = "0" ]; then
echo "hiluo-file-resize is running"
RETVAL=0
else
echo "hiluo-file-resize is not running"
RETVAL=0
fi
fi
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
status
;;
*)
echo "Usage $0 {start|stop|restart|status}"
RETVAL=1
esac
exit $?
注意将dir改成实际的部署路径。这个可以再改进，自动识别当前路径。
8.效果如下
8.1上传页面
8.2上传成功页面
8.3缩略图页面
文件打包下载见附件！
参考：
http://webpy.org/cookbook/storeupload/
http://webpy.org/cookbook/fastcgi-nginx
http://www.pythonclub.org/python-basic/string
http://hi.baidu.com/thinkinginlamp/blog/item/4b61e9241f08820f4c088d95.html