【原创】用python连接thrift Server 去执行sql的问题总汇
场景:python和现有产品的结合和应用——python的前瞻性调研
环境:centos7
0、首先确保安装了python和pyhive,下面是连接代码:
#!/usr/bin/env python # -*- coding: utf-8 -*- import sys from pyhive import hive from TCLIService.ttypes import TOperationState def pyhiveexesql(sql): cursor = None try: cursor = hive.connect(host='10.19.12.20', port=10015, username='xxx').cursor()
cursor.execute(sql, async=True) status = cursor.poll().operationState while status in (TOperationState.INITIALIZED_STATE, TOperationState.RUNNING_STATE): logs = cursor.fetch_logs() for message in logs: print message # If needed, an asynchronous query can be cancelled at any time with: # cursor.cancel() status = cursor.poll().operationState print cursor.fetchall() except Exception, e: print '%s' % (e.message) finally: cursor.close() if __name__ == '__main__': pyhiveexesql('SELECT * FROM my_awesome_data LIMIT 10')
问题一:ImportError: No module named sasl
解决:找不到saal就装嘛,执行:sudo pip install sasl,于是爆出第二个问题
问题二:Error:command 'gcc' failed with exit status 1
解决:网上说安装sasl之前有一些前置支撑包,好吧有安装了一些,但还是无动于衷,于是疯狂找了一天终于找到了解决方案:
执行:sudo yum install libffi-devel;sudo yum install libgsasl-devel;sudo yum install libmemcached-devel;
好吧,问题一解决了,但是又爆出问题三:
问题三:ImportError: No module named thrift_sasl
解决方案:sudo yum -y install easy_install; sudo easy_install thrift; sudo pip install thrift_sasl;
好吧问题二和问题三解决了,但是又爆出问题四:
问题四:pyhive.exc.OperationalError: TFetchResultsResp(status=TStatus(errorCode=0, errorMessage=u'Expected state FINISHED, but found ERROR'
具体问题:
![](http://img.e-com-net.com/image/info8/b8d97b5613f94ed2ba791cad57d0b2ed.gif)
![](http://img.e-com-net.com/image/info8/2f88dd3f1cd145f59c0e47b51acdbd4b.gif)
[hfb@192 ~]$ python Desktop/pyhive4.py Traceback (most recent call last): File "Desktop/pyhive4.py", line 31, inprint cursor.fetchall() File "/usr/lib/python2.7/site-packages/pyhive/common.py", line 145, in fetchall one = self.fetchone() File "/usr/lib/python2.7/site-packages/pyhive/common.py", line 105, in fetchone self._fetch_while(lambda: not self._data and self._state != self._STATE_FINISHED) File "/usr/lib/python2.7/site-packages/pyhive/common.py", line 45, in _fetch_while self._fetch_more() File "/usr/lib/python2.7/site-packages/pyhive/hive.py", line 318, in _fetch_more _check_status(response) File "/usr/lib/python2.7/site-packages/pyhive/hive.py", line 421, in _check_status raise OperationalError(response) pyhive.exc.OperationalError: TFetchResultsResp(status=TStatus(errorCode=0, errorMessage=u'Expected state FINISHED, but found ERROR', sqlState=None, infoMessages=[u'*org.apache.hive.service.cli.HiveSQLException:Expected state FINISHED, but found ERROR:15:14', u'org.apache.hive.service.cli.operation.Operation:assertState:Operation.java:161', u'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:getNextRowSet:SparkExecuteStatementOperation.scala:107', u'org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:220', u'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:685', u'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:454', u'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:672', u'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', u'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', u'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', u'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', u'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', u'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285', u'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149', u'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624', u'java.lang.Thread:run:Thread.java:748'], statusCode=3), results=None, hasMoreRows=None)
解决方案:表不存在,哈哈,这个错误信息不明显啊
完。。。