Redis 连接编码问题

Redis 连接编码问题

不编码情况下，UnicodeDecodeError

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/python3/lib/python3.4/site-packages/redis/client.py", line 863, in get
    return self.execute_command('GET', name)
  File "/usr/local/python3/lib/python3.4/site-packages/redis_py_cluster-1.0.0-py3.4.egg/rediscluster/utils.py", line 85, in inner
    _keys = set(sum([list(obj.keys()) for obj in objs],[]))  
  File "/usr/local/python3/lib/python3.4/site-packages/redis_py_cluster-1.0.0-py3.4.egg/rediscluster/client.py", line 293, in execute_command
  File "/usr/local/python3/lib/python3.4/site-packages/redis/client.py", line 577, in parse_response
    response = connection.read_response()
  File "/usr/local/python3/lib/python3.4/site-packages/redis/connection.py", line 569, in read_response
    response = self._parser.read_response()
  File "/usr/local/python3/lib/python3.4/site-packages/redis/connection.py", line 266, in read_response
    response = response.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

代码

>>> from redis import StrictRedis
>>> r = StrictRedis('localhost', 6380)
>>> r.set('hello', '你好')
True
>>> r.get('hello')
b'\xe4\xbd\xa0\xe5\xa5\xbd'

可以看见，字符串输入被编码成utf8存储在Redis里了。而取出来的时候还是被编码后的bytes，需要显示的decode才能变成字符串。

解决方案

Redis建立连接时有两个参数，一个是encoding指定编码，默认是utf8。一个是decode_responses，默认为False，如果是True，会以encoding方式解码，然后返回字符串。如果是字符串，就根据encoding编码成bytes。

>>> from redis import StrictRedis
>>> r = StrictRedis('localhost', 6380, encoding='utf8', decode_responses=True)
>>> r.set('hello', '你好')
True
>>> r.get('hello')
'你好'

压缩 (gzip, zlib, lz4) 情况下，UnicodeDecodeError

依然报上面的错

解决方案

decode_responses 设为 False

连接 redis 时带上 decode_responses=False 取消对返回结果的 decode('utf-8') 操作

>>import redis
>>import gzip
>>rdb = redis.Redis(host='127.0.0.1',6379,db=0,decode_responses=False)
>>rdb.set('test.zip',gzip.compress(b'xxxxxxxxxxxxx'))
>>rdb.get('test.zip')
b'\x1f\x8b\x08\x00\xd5;\xe1V\x02\xff\xab\xa8@\x02\x00\xd6\x82q\xe4\r\x00\x00\x00'

带来的弊端的返回的结果是byte类型，要自行decode

网上还有人推荐修改源码，我没尝试过，直接粘贴过来的 ，但也是一种方案嘛。

位置在 python安装目录/lib/python3.4/site-packages/redis/connection.py 266行
注：可以先将egg包解压，然后将egg包更个名
如此未用gzip压缩的返回的是utf-8，而经过gzip压缩过的返回的是byte

修改 redis 源码

                elif byte == '+':
            pass
        # int value
        elif byte == ':':
            response = long(response)
        # bulk response
        elif byte == '$':
            length = int(response)
            if length == -1:
                return None
            response = self._buffer.read(length)
        # multi-bulk response
        elif byte == '*':
            length = int(response)
            if length == -1:
                return None
            response = [self.read_response() for i in xrange(length)]
        if isinstance(response, bytes) and self.encoding:
            #此处加个异常处理就可以了
            try:
                response = response.decode()
            except :
                pass
        return response

结果

>>import redis
>>import gzip
>>rdb = redis.Redis(host='127.0.0.1',6379,db=0,decode_responses=False)
>>rdb.set('test.zip',gzip.compress(b'xxxxxxxxxxxxx'))
>>rdb.get('test.zip')
b'\x1f\x8b\x08\x00\xd5;\xe1V\x02\xff\xab\xa8@\x02\x00\xd6\x82q\xe4\r\x00\x00\x00'
>>rdb.set('test.str','xxxx')
>>rdb.get('test.str')
'xxxx'