Win7平台Python3使用impyla连接Hive遇到的坑
环境硬件配置及Hadoop，Hive版本
此博客置顶文章中有
安装步骤
pip install pure-saslLooking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting pure-sasl
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/16/83/30eaf3765de898083
75a8358f9c15d45a3dd44ed26be991471abc0b4480b/pure_sasl-0.5.1-py2.py3-none-any.whlInstalling collected packages: pure-sasl
Successfully installed pure-sasl-0.5.1
pip install thrift_sasl==0.2.1 --no-depsLooking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting thrift_sasl==0.2.1
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/80/36/16dfe92d32df63cc2
b7b7be8d0e4a736617b7e52daaa7f83ae386a89d179/thrift_sasl-0.2.1.tar.gz
Installing collected packages: thrift-sasl
Running setup.py install for thrift-sasl … done
Successfully installed thrift-sasl-0.2.1
pip install thrift==0.9.3Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting thrift==0.9.3
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/58/35e3f0cd290039ff8
62c2c9d8ae8a76896665d70343d833bdc2f748b8e55/thrift-0.9.3.tar.gz
Installing collected packages: thrift
Running setup.py install for thrift … done
Successfully installed thrift-0.9.3
pip install impylaLooking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting impyla
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/80/93/f0d226061ee4679d5
b593c88c7b2e9e077a271c799d29facf31bf03666c1/impyla-0.14.1.tar.gz (151kB)
40% |█████████████ | 61kB 1.3MB/s eta 0:00:01
47% |███████████████ | 71kB 1.1MB/s eta 0:00:
53% |█████████████████ | 81kB 1.1MB/s eta 0:0
60% |███████████████████▌ | 92kB 1.2MB/s eta
67% |█████████████████████▌ | 102kB 1.1MB/s e
74% |████████████████████████ | 112kB 1.1MB/s
80% |██████████████████████████ | 122kB 759kB
87% |████████████████████████████ | 133kB 786
94% |██████████████████████████████ | 143kB 8
100% |████████████████████████████████| 153k
B 1.0MB/s
Requirement already satisfied: six in c:\program files\python36\lib\site-package
s (from impyla) (1.11.0)
Collecting bitarray (from impyla)
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e2/1e/b93636ae36d08d0ee
3aec40b08731cc97217c69db9422c0afef6ee32ebd2/bitarray-0.8.3.tar.gz
Collecting thriftpy>=0.3.5 (from impyla)
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f4/19/cca118cf7d2087310
dbc8bd70dc7df0c1320f2652873a93d06d7ba356d4a/thriftpy-0.3.9.tar.gz (208kB)
39% |████████████▌ | 81kB 875kB/s eta 0:00:01
44% |██████████████ | 92kB 984kB/s eta 0:00:0
49% |████████████████ | 102kB 820kB/s eta 0:0
54% |█████████████████ | 112kB 820kB/s eta 0:
59% |███████████████████ | 122kB 820kB/s eta
63% |████████████████████▌ | 133kB 729kB/s et
68% |██████████████████████ | 143kB 937kB/s e
73% |███████████████████████▌ | 153kB 937kB/s
78% |█████████████████████████ | 163kB 820kB/
83% |███████████████████████████ | 174kB 937k
88% |████████████████████████████▌ | 184kB 93
93% |██████████████████████████████ | 194kB 9
98% |███████████████████████████████▌| 204kB
100% |████████████████████████████████| 215k
B 1.1MB/s
Requirement already satisfied: ply<4.0,>=3.4 in c:\program files\python36\lib\si
te-packages (from thriftpy>=0.3.5->impyla) (3.11)
thrift-sasl 0.2.1 requires sasl>=0.2.1, which is not installed.
Installing collected packages: bitarray, thriftpy, impyla
Running setup.py install for bitarray … done
Running setup.py install for thriftpy … done
Running setup.py install for impyla … done
Successfully installed bitarray-0.8.3 impyla-0.14.1 thriftpy-0.3.9
执行数据库连接后，出现问题
ThriftParserError: ThriftPy does not support generating module with path in protocol ‘c’
定位到 \Lib\site-packages\thriftpy\parser\parser.py的
if url_scheme == '':
with open(path) as fh:
data = fh.read()
elif url_scheme in ('http', 'https'):
data = urlopen(path).read()
else:
raise ThriftParserError('ThriftPy does not support generating module '
'with path in protocol \'{}\''.format(
url_scheme))
更改为
if url_scheme == '':
with open(path) as fh:
data = fh.read()
elif url_scheme in ('c', 'd','e','f''):
with open(path) as fh:
data = fh.read()
elif url_scheme in ('http', 'https'):
data = urlopen(path).read()
else:
raise ThriftParserError('ThriftPy does not support generating module '
'with path in protocol \'{}\''.format(
url_scheme))
执行数据库连接后，再次出现问题
TypeError: can’t concat str to bytes
定位到错误的最后一条，在init.py第94行
...
header = struct.pack(">BI", status, len(body))
self._trans.write(header + body)
...修改为
...
header = struct.pack(">BI", status, len(body))
if(type(body) is str):
body = body.encode()
self._trans.write(header + body)
...执行连接 成功
最终模块
| 模块 | 版本 |
|---|---|
| pure_sasl | 0.5.1 |
| thrift_sasl | 0.2.1 |
| thrift | 0.9.3 |
| bitarray | 0.8.3 |
| thriftpy | 0.3.9 |
| impyla | 0.14.1 |
连接代码
from impala.dbapi import connect
from impala.util import as_pandas
conn = connect(host='***', port=10000, auth_mechanism='PLAIN', user='***', password='***', database='***')
cursor = conn.cursor()
cursor.execute('show databases')
print(as_pandas(cursor))遇到的问题
除去以上两个问题还有
| ‘TSocket’ object has no attribute ‘isOpen’ |
|---|
| Can’t connect to unsecured hive. SASL error: TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2 |
| TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL( -4): no mechanism available: Unable to find a callback: 2 |
| ‘No protocol version header’) thriftpy.protocol.exc.TProtocolException: TProtocolException(type=4) |
以上问题均可以
pip uninstall卸载所有相关模块之后， 使用本文的顺序和版本号安装来保证运行正确
参考文章
| https://blog.csdn.net/a6822342/article/details/80844072 |
|---|
| https://github.com/cloudera/impyla/issues/268 |
| https://github.com/cloudera/impyla/issues/165 |
| https://github.com/dropbox/PyHive/issues/161 |
| https://github.com/cloudera/impyla/issues/149 |
| https://github.com/dropbox/PyHive/issues/32 |
| https://github.com/eleme/thriftpy/issues/234 |
| https://github.com/cloudera/impyla/issues/267 |
| https://blog.csdn.net/qq_24908345/article/details/80595948 |
| https://blog.csdn.net/sinolover/article/details/77714648 |
| https://github.com/eleme/thriftpy/issues/234 |
| https://www.cnblogs.com/angellst/p/7510486.html |
| https://www.cnblogs.com/tomato0906/articles/6057335.html |