python3 连接impala和Hive

CentOS7 版本的安装:

1.设置repo文件:
#wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo


2.安装python3.6版本:

# yum -y install python36 python36-devel python36-pip 
3.升级pip版本:
# pip3 -V
pip 8.1.2 from /usr/lib/python3.6/site-packages (python 3.6)

# pip3 install --upgrade pip
# pip3.6  -V
pip 19.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

# pip -V
pip 19.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)


# pip install impyla
Collecting impyla
  Downloading https://files.pythonhosted.org/packages/fb/e4/f2a71cb427cf5eb9d0dec0026220bbfe3efafe67b676e2b96ea5a5d3bff2/impyla-0.15.0.tar.gz (233kB)
     |████████████████████████████████| 235kB 623kB/s 
Collecting six (from impyla)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting bitarray (from impyla)
  Downloading https://files.pythonhosted.org/packages/f2/f1/d5c824741a19d2386374cbc62caada0e90bcf6ee4212490b272a8054428d/bitarray-0.9.2.tar.gz
Collecting thrift>=0.9.3 (from impyla)
  Downloading https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz (52kB)
     |████████████████████████████████| 61kB 11.2MB/s 
Collecting thriftpy2==0.4.0 (from impyla)
  Downloading https://files.pythonhosted.org/packages/d5/bd/8843470763354f509ed7ccec08722f405c9a3b714c51c9866cca4cfe55c7/thriftpy2-0.4.0.tar.gz (293kB)
     |████████████████████████████████| 296kB 15.6MB/s 
Collecting ply<4.0,>=3.4 (from thriftpy2==0.4.0->impyla)
  Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
     |████████████████████████████████| 51kB 10.6MB/s 
Installing collected packages: six, bitarray, thrift, ply, thriftpy2, impyla
  Running setup.py install for bitarray ... done
  Running setup.py install for thrift ... done
  Running setup.py install for thriftpy2 ... done
  Running setup.py install for impyla ... done
Successfully installed bitarray-0.9.2 impyla-0.15.0 ply-3.11 six-1.12.0 thrift-0.11.0 thriftpy2-0.4.0

# pip install impalacli

依赖的系统软件包:
# rpm -qa | grep -i sasl
libgsasl-1.8.0-8.el7.x86_64
cyrus-sasl-lib-2.1.26-23.el7.x86_64
cyrus-sasl-plain-2.1.26-23.el7.x86_64
cyrus-sasl-2.1.26-23.el7.x86_64
libgsasl-devel-1.8.0-8.el7.x86_64
saslwrapper-devel-0.16-5.el7.x86_64
cyrus-sasl-gssapi-2.1.26-23.el7.x86_64
cyrus-sasl-devel-2.1.26-23.el7.x86_64
saslwrapper-0.16-5.el7.x86_64
# yum -y install saslwrapper  cyrus-sasl cyrus-sasl-devel libgsasl libgsasl-devel gcc gcc-c++
# pip install thrift_sasl

--版本查询:
# pip list
Package    Version
---------- -------
bitarray   0.9.2  
impyla     0.15.0 
pip        19.1   
ply        3.11   
setuptools 39.2.0 
six        1.12.0 
thrift     0.11.0 
thriftpy2  0.4.0  
-- 
cli-helpers    1.0.2  
Click          7.0    
docopt         0.6.2  
impalacli      0.1.5  
prompt-toolkit 1.0.15 
Pygments       2.2.0  
tabulate       0.8.3  
terminaltables 3.1.0 
thriftpy       0.3.9   
wcwidth        0.1.7  
--
sasl           0.2.1  
thrift-sasl    0.3.0  

impyla的使用:


windows 10 版本的impyla安装:
windows 10 版本下需要安装vc++
 https://visualstudio.microsoft.com/downloads/
 报错信息:
building 'bitarray._bitarray' extension
  error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/
  ----------------------------------------
  ERROR: Failed building wheel for bitarray

类似需要的项目有word_count 和scrapy等依赖于c++的。
https://github.com/amueller/word_cloud/
需要安装软件: visualcppbuildtools_full.exe
此软件从互联网下载软件包比较慢。

C:\Users\Administrator>pip3 install   impyla
Collecting impyla
Requirement already satisfied: six in c:\dev\python36\lib\site-packages (from impyla) (1.11.0)
Collecting bitarray (from impyla)
  Using cached https://files.pythonhosted.org/packages/f2/f1/d5c824741a19d2386374cbc62caada0e90bcf6ee4212490b272a8054428d/bitarray-0.9.2.tar.gz
Requirement already satisfied: thrift>=0.9.3 in c:\dev\python36\lib\site-packages (from impyla) (0.11.0)
Requirement already satisfied: thriftpy2==0.4.0; python_version >= "3.0" in c:\dev\python36\lib\site-packages (from impyla) (0.4.0)
Requirement already satisfied: ply<4.0,>=3.4 in c:\dev\python36\lib\site-packages (from thriftpy2==0.4.0; python_version >= "3.0"->impyla) (3.11)
Building wheels for collected packages: bitarray
  Building wheel for bitarray (setup.py) ... done
  Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\ba\f7\bf\917a0b12de06d195ca40cb0406cb58d4f21fe2e74f3571fb9a
Successfully built bitarray
Installing collected packages: bitarray, impyla
Successfully installed bitarray-0.9.2 impyla-0.15.0

C:\Users\Administrator>pip list | findstr impyla
impyla                      0.15.0

C:\Users\Administrator>pip list | findstr /i "impyla bitarray thrift thriftpy2 ply"
bitarray                    0.9.2
impyla                      0.14.1
ply                         3.11
thrift                      0.11.0
thriftpy                    0.3.9
thriftpy2                   0.4.0
原因: ERROR: impalacli 0.1.5 has requirement impyla==0.14.1, but you'll have impyla 0.15.0 which is incompatible.



--windows 版本查询版本:
C:\Users\Administrator>pip list | findstr /i "twist scrapy wordcloud"
Scrapy                      1.6.0
Twisted                     19.2.0
wordcloud                   1.5.0

C:\Users\Administrator>pip list | findstr /i "tensorflow pyqt opencv"
opencv-python               4.1.0.25
PyQt5                       5.9
tensorflow                  1.12.0

-- 常见的数据库连接器:
C:\Users\Administrator>pip list | findstr /i "redis mongo elasticsearch mysql"
elasticsearch               7.0.0
mysql-connector             2.1.6
mysql-connector-python      8.0.16
pymongo                     3.8.0
PyMySQL                     0.9.3
redis                       3.2.1
redis-completion            0.5.0


Hive的连接其他连接方式:
# pip install pyhive[hive]
Collecting pyhive[hive]
  Downloading https://files.pythonhosted.org/packages/4e/26/de91125c0d9e8947d48f387f4d1f2e7a22aa92a30771ad02f63a5653361b/PyHive-0.6.1.tar.gz (41kB)
     |████████████████████████████████| 51kB 446kB/s 
Collecting future (from pyhive[hive])
  Downloading https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz (829kB)
     |████████████████████████████████| 829kB 1.7MB/s 
Collecting python-dateutil (from pyhive[hive])
  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
     |████████████████████████████████| 235kB 16.4MB/s 
Requirement already satisfied: sasl>=0.2.1 in /usr/local/lib64/python3.6/site-packages (from pyhive[hive]) (0.2.1)
Requirement already satisfied: thrift>=0.10.0 in /usr/local/lib64/python3.6/site-packages (from pyhive[hive]) (0.11.0)
Requirement already satisfied: thrift_sasl>=0.1.0 in /usr/local/lib/python3.6/site-packages (from pyhive[hive]) (0.3.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/site-packages (from python-dateutil->pyhive[hive]) (1.12.0)
Installing collected packages: future, python-dateutil, pyhive
  Running setup.py install for future ... done
  Running setup.py install for pyhive ... done
Successfully installed future-0.17.1 pyhive-0.6.1 python-dateutil-2.8.0


# pip install pyhive[presto]
Requirement already satisfied: pyhive[presto] in /usr/local/lib/python3.6/site-packages (0.6.1)
Requirement already satisfied: future in /usr/local/lib/python3.6/site-packages (from pyhive[presto]) (0.17.1)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/site-packages (from pyhive[presto]) (2.8.0)
Collecting requests>=1.0.0 (from pyhive[presto])
  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
     |████████████████████████████████| 61kB 536kB/s 
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/site-packages (from python-dateutil->pyhive[presto]) (1.12.0)
Collecting idna<2.9,>=2.5 (from requests>=1.0.0->pyhive[presto])
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
     |████████████████████████████████| 61kB 16.3MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests>=1.0.0->pyhive[presto])
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
     |████████████████████████████████| 143kB 1.8MB/s 
Collecting certifi>=2017.4.17 (from requests>=1.0.0->pyhive[presto])
  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
     |████████████████████████████████| 163kB 11.0MB/s 
Collecting urllib3<1.25,>=1.21.1 (from requests>=1.0.0->pyhive[presto])
  Downloading https://files.pythonhosted.org/packages/df/1c/59cca3abf96f991f2ec3131a4ffe72ae3d9ea1f5894abe8a9c5e3c77cfee/urllib3-1.24.2-py2.py3-none-any.whl (131kB)
     |████████████████████████████████| 133kB 12.7MB/s 
Installing collected packages: idna, chardet, certifi, urllib3, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.2


参考:
https://github.com/cloudera/impyla

https://github.com/dropbox/PyHive

 

你可能感兴趣的:(Python)