Python脚本通过unixODBC驱动访问Greenplum(4.3.8.2)数据库安装指导

本篇文档主要用来描述：

1.    搭建unixODBC驱动，用来通过odbc方式访问数据库

2.    搭建Pyodbc驱动，用来使用Python脚本通过系统的odbc方式来对数据库进行操作

3.    这种架构的好处：

l  通过odbc访问数据库的性能要好于jdbc的方式

l  通过Python脚本开发，后续可以通过调度平台来执行Python执行数据分析，比如设置定时任务，周期性地执行Python脚本

 

安装前准备

（1）操作系统(系统上面要安装一些必备的开发工具(比如gcc等))

[root@CDHA ~]# cat /etc/redhat-release

CentOS release 6.7 (Final)

 

[root@CDHA ~]$ python
Python 2.6.2 (r262:71600, May 12 2009, 15:34:31)
[GCC 4.1.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

 

（2）安装所需的软件包

greenplum-connectivity-4.3.8.2-build-1-RHEL5-x86_64.zip

--GP官网下载，GP的JDBC和ODBC驱动

 

pyodbc-3.0.10.tar.gz                        

 --Python连接GP需要pyodbc驱动包

 

unixODBC-2.2.12.tar.gz               

--unixODBC的驱动管理器

 

（3）将上面的包上传到搭建环境的CDHA服务器上面，比如/software/

 

安装GP驱动包

1.    解压greenplum-connectivity-4.3.8.2-build-1-RHEL5-x86_64.zip

unzip greenplum-connectivity-4.3.8.2-build-1-RHEL5-x86_64.zip

 

2.    执行解压后得到greenplum-connectivity-4.3.8.2-build-1-RHEL5-x86_64.bin可执行文件

bash greenplum-connectivity-4.3.8.2-build-1-RHEL5-x86_64.bin

 略部分内容

********************************************************************

    Do you accept the EMC Connectivity license agreement? [yes | no]

********************************************************************

 

yes     ---------同意许可

********************************************************************

Providethe installation path for Greenplum Connectivity or press ENTER to

Accept the default installation path:greenplum-connectivity-4.3.8.2-build-1

********************************************************************

********************************************************************

InstallGreenplum Connectivity into? [yes | no]

********************************************************************

 

yes      ----------------保持默认的安装路径，你也可以自由指定安装路径

 

********************************************************************

/usr/local/greenplum-connectivity-4.3.8.2-build-1 does not exist.

Create/usr/local/greenplum-connectivity-4.3.8.2-build-1 ? [ yes | no ]

(Selectingno will exit the installer)

********************************************************************

 

yes      ----------------创建安装目录           

 

Extractingproduct to /usr/local/greenplum-connectivity-4.3.8.2-build-1

 

********************************************************************

Installationcomplete.

GreenplumConnectivity is installed in /usr/local/greenplum-connectivity-4.3.8.2-build-1

************************************************************************

 

3.    配置Greenplum DB数据库驱动

查看安装目录时，如下：

[root@CDHA software]# ll /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc

total 24

drwxr-xr-x 3 gpadmin gpadmin 4096 May 10 13:34 psqlodbc-08.02.0400

drwxr-xr-x 6 gpadmin gpadmin 4096 May 10 13:37 psqlodbc-08.02.0500

drwxr-xr-x 3 gpadmin gpadmin 4096 May 10 13:37 psqlodbc-08.03.0400

drwxr-xr-x 3 gpadmin gpadmin 4096 May 10 13:38 psqlodbc-08.04.0200

drwxr-xr-x 3 gpadmin gpadmin 4096 May 10 13:38 psqlodbc-09.00.0200

drwxr-xr-x 3 gpadmin gpadmin 4096 May 10 13:38 psqlodbc-09.02.0100

[root@CDHA software]#

我们会看到有好几个版本的驱动，我们可以选择psqlodbc-08.02.0500版本的，再查看如下目录：

[root@CDHA software]# ll /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/

total 48

drwxr-xr-x 2 gpadmin gpadmin  4096 May 10 13:37 datadirect-51sp2_64

drwxr-xr-x 2 gpadmin gpadmin  4096 May 10 13:37 datadirect-52_64

drwxr-xr-x 2 gpadmin gpadmin  4096 May 10 13:37 datadirect-53sp2_64

-rwxr-xr-x 1 gpadmin gpadmin 25746 May 10 13:36 license.txt

-rwxr-xr-x 1 gpadmin gpadmin  1383 May 10 13:36 readme.txt

drwxr-xr-x 3 gpadmin gpadmin  4096 Jul  6 11:38 unixodbc-2.2.12

 

同样我们可以看到驱动管理器。

鉴于GP基于Postgresql8.2版本，我们这里面选择驱动为psqlodbc-08.02.0500，驱动管理器选择为datadirect-52_64。

所以，我们修改greenplum_connectivity_path.sh文件中的内容：

GP_ODBC_DRIVER=psqlodbc-08.02.0500          --值与实际目录名称相同

GP_ODBC_DRIVER_MANAGER=datadirect-52_64    --值与实际目录名称相同

 

注：该文件默认权限位444，是不允许编辑的，你可以手动修改文件的权限，也可以修改整个安装目录的权限位755，如下：

chmod -R 755 /usr/local/greenplum-connectivity-4.3.8.2-build-1

 

保存greenplum_connectivity_path.sh后，要记得source，使环境变量生效，如下：

source greenplum_connectivity_path.sh

 

安装unixODBC驱动

1.    编译和安装unixODBC驱动包

tar -zxvf unixODBC-2.2.12.tar.gz

./configure --prefix=/etc/unixODBC --enable-fdb  --disable-gui

make

make install

2.    查看unixODBC安装目录

[root@CDHA software]# ll /etc/unixODBC/

total 16

drwxr-xr-x 2 root root 4096 Jul  5 23:19 bin

drwxr-xr-x 3 root root 4096 Jul  6 10:09 etc

drwxr-xr-x 2 root root 4096 Jul  5 23:19 include

drwxr-xr-x 2 root root 4096 Jul  6 11:47 lib

 

3.    编辑unixODBC的etc目录下面的两个配置文件，如下：

[root@CDHA unixODBC]# cat /etc/unixODBC/etc/odbc.ini

[GreenplumDSN]

Driver =Greenplum   ----值要和/etc/unixODBC/etc/odbcinst.ini中名字一致

Trace = 1

Debug=1

Database = zhangyun_db_safe    ----GP数据库名

Servername = 192.168.1.24   ----GP的IP地址

UserName = zhangyun    ----GP用户名

Password = xxxxxx       ----GP用户密码

Port = 5432                ----GP访问端口号

ReadOnly = No

RowVersioning = No

DisallowPremature = No

ShowSystemTables = Yes

ShowOidColumn = No

FakeOidIndex = No

useDeclareFetch = 1

Fetch = 4096

UpdatableCursors = Yes

Protocol = 7.4-1

 

[root@CDHA unixODBC]# cat /etc/unixODBC/etc/odbcinst.ini

[Greenplum]

Description=PostgreSQL driver forGreenplum

Driver=/usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/unixodbc-2.2.12/psqlodbcw.so   ----GP的ODBC驱动

UsageCount=1

FileUsage=1

 

4.    使用isql测试

[root@CDHA unixODBC]# isql GreenplumDSN zhangyun xxxxxx

+---------------------------------------+

| Connected!                            |

|                                       |

| sql-statement                         |

| help [tablename]                      |

| quit                                  |

|                                       |

+---------------------------------------+

SQL> select count(1) from test_bigtable;

+---------------------+

| count               |

+---------------------+

| 1791144834          |

+---------------------+

SQLRowCount returns -1

1 rows fetched

SQL> select user;  

+-----------------------------------------------------------------+

| current_user                                                    |

+-----------------------------------------------------------------+

| zhangyun                                                      |

+-----------------------------------------------------------------+

SQLRowCount returns -1

1 rows fetched

SQL>

 

注：如果你在执行isql时，出现如下情况：

[root@CDHA unixODBC]# isql GreenplumDSN

[ISQL]ERROR: Could not SQLConnect

 

这个问题很大情况下是你没有source文件greenplum_connectivity_path.sh导致的，执行source greenplum_connectivity_path.sh文件后，再执行就OK了，最好的办法是将source该文件加入到系统环境变量中。

如果还是不可以的话，请加入-v选项查看详细信息，也可参考最后的错误的解决方法。

 

安装pyodbc驱动

1.    编译和安装pyodbc驱动

在编译之前说明一下，如下的一些库需要安装好，如下：

[root@CDHA pyodbc-3.0.10]# yum install gcc-c*

[root@CDHA pyodbc-3.0.10]# yum install compat-gcc-34-c++

[root@CDHA pyodbc-3.0.10]# yum install unixODBC-devel

[root@CDHA pyodbc-3.0.10]# yum install python-devel

如果还有其他库没有安装，请执行yum来进行安装。

下面开始编译pyodbc

tar -zxvf pyodbc-3.0.10.tar.gz

cd pyodbc-3.0.10

python setup.py build

python setup.py install

 

2.    查看pyodbc安装目录

[root@CDHA pyodbc-3.0.10]# ll /usr/lib64/python2.6/site-packages/pyodbc*

-rwxr-xr-x 1 root root    913 Jul  6 14:34 /usr/lib64/python2.6/site-packages/pyodbc-3.0.10-py2.6.egg-info

-rwxr-xr-x 1 root root 391676 Jul  6 14:34 /usr/lib64/python2.6/site-packages/pyodbc.so

 

测试python脚本

1.    准备python测试脚本，如下：

[zhangyun@CDHA ~]$ cat helloworld.py

#!/usr/bin/python

#-*- encoding: utf-8 -*-

####################################################################

# name: helloworld.py   

# describe: 测试python访问Greenplum数据库

########################################################################

import pyodbc

import sys

reload(sys)

sys.setdefaultencoding('utf8')

 

class GreenplumTest:

    debug = 1

    def __init__(self,dbinfo):

        self.UID = dbinfo[1]

        self.PWD = dbinfo[2]

        odbcinfo    ='DSN=%s;UID=%s;PWD=%s'%(dbinfo[0],dbinfo[1],dbinfo[2])

        self.cnxn   =pyodbc.connect(odbcinfo,autocommit=True,ansi=True)

        self.cursor =self.cnxn.cursor()

 

    def __del__(self):

        if self.cursor:

            self.cursor.close()

        if self.cnxn:

            self.cnxn.close()

 

    def _printinfo(self,msg):

        print"%s"%(msg)

        print "\n"

 

    def testsql(self):

        # 类似的业务逻辑，可以放到sql中执行

        # 示例：创建表，插入数据

        sql_1 = '''

          drop table if exists helloworld;

          create table helloworld(id int,name text) distributed by (id);

          insert into helloworld values(1,'Spark'),(2,'Hadoop'),(3,'Apache');

          '''

        self.cursor.execute(sql_1.strip())

       

        #查询结果，并返回

        sql_2 = '''

          select * from helloworld;

          ''' 

        self.cursor.execute(sql_2.strip())             

        row = self.cursor.fetchall()

        return row 

 

#Main

def main():

    # 检查传入参数个数    

    if len(sys.argv) < 4 :

        print 'usage: python GreenplumDSN username password\n'

        sys.exit(1)

 

    # 定义连接GP的信息     

    dbinfo = []

    dbinfo.append(sys.argv[1])

    dbinfo.append(sys.argv[2])

    dbinfo.append(sys.argv[3])

 

    GPT= GreenplumTest(dbinfo)

    ret = GPT.testsql()

    return ret

 

if __name__ == '__main__':

    sys.exit(main())

 

2.    测试过程：

python helloworld.py GreenplumDSN zhangyun xxxxxx

上面执行后返回结果如下：

[(3, 'Apache'), (1, 'Spark'), (2, 'Hadoop')]

 

3.    我们登录GP数据库查看数据

[gpadmin@CDHA ~]$ psql -d zhangyun_db_safe -U zhangyun -h CDHA

psql (8.2.15)

Type "help" for help.

 

zhangyun_db_safe=# select * from helloworld ;

 id |  name 

----+--------

  2 | Hadoop

  1 | Spark

  3 | Apache

(3 rows)

 

zhangyun_db_safe=# 

 

 

 

问题汇总：

1.       如果unixODBC安装好后，在使用isql时，显示如下错误

/usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/unixodbc-2.2.12/psqlodbcw.so文件找不到，如：

 

isql GreenplumDSN -v

 

[01000][unixODBC][Driver Manager]Can't open lib '/usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/unixodbc-2.2.12/psqlodbcw.so' : file not found

[ISQL]ERROR: Could not SQLConnect

 

解决办法：

首先使用ldd查看缺失什么文件

ldd /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/unixodbc-2.2.12/psqlodbcw.so

 

linux-vdso.so.1 =>  (0x00007ffdd9106000)

libpq.so.5 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libpq.so.5 (0x00002addb8e3f000)

         libpthread.so.0 => /lib64/libpthread.so.0 (0x00002addb908c000)

         libodbcinst.so.1 => not found

         libodbc.so.1 => not found

         libc.so.6 => /lib64/libc.so.6 (0x00002addb92aa000)

         libssl.so.0.9.8 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libssl.so.0.9.8 (0x00002addb963e000)

         libcrypto.so.0.9.8 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libcrypto.so.0.9.8 (0x00002addb9893000)

         libgssapi_krb5.so.2 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libgssapi_krb5.so.2 (0x00002addb9c26000)

         libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002addb9e50000)

         libldap_r-2.3.so.0 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libldap_r-2.3.so.0 (0x00002addba088000)

         /lib64/ld-linux-x86-64.so.2 (0x0000003f8bc00000)

         libdl.so.2 => /lib64/libdl.so.2 (0x00002addba2dc000)

         libkrb5.so.3 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libkrb5.so.3 (0x00002addba4e1000)

         libk5crypto.so.3 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libk5crypto.so.3 (0x00002addba76b000)

         libcom_err.so.3 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libcom_err.so.3 (0x00002addba990000)

         libkrb5support.so.0 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libkrb5support.so.0 (0x00002addbab95000)

         libresolv.so.2 => /lib64/libresolv.so.2 (0x00002addbad9c000)

         libfreebl3.so => /lib64/libfreebl3.so (0x00002addbafb7000)

         liblber-2.3.so.0 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/liblber-2.3.so.0 (0x00002addbb1ba000)

 

那么我们就创建缺失的文件，如下：

ln -s /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/unixodbc-2.2.12/libodbcinst.so.1 /lib64/libodbcinst.so.1 

ln -s /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/unixodbc-2.2.12/libodbc.so.1 /lib64/libodbc.so.1

 

 

再次查看时就没有问题了，如下：

ldd /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/unixodbc-2.2.12/psqlodbcw.so

 

         linux-vdso.so.1 =>  (0x00007ffcc97d4000)

         libpq.so.5 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libpq.so.5 (0x00002b10ed9ee000)

         libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b10edc3b000)

         libodbcinst.so.1 => /lib64/libodbcinst.so.1 (0x00002b10ede58000)

         libodbc.so.1 => /lib64/libodbc.so.1 (0x00002b10ee071000)

         libc.so.6 => /lib64/libc.so.6 (0x00002b10ee300000)

         libssl.so.0.9.8 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libssl.so.0.9.8 (0x00002b10ee694000)

         libcrypto.so.0.9.8 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libcrypto.so.0.9.8 (0x00002b10ee8e9000)

         libgssapi_krb5.so.2 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libgssapi_krb5.so.2 (0x00002b10eec7c000)

         libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b10eeea6000)

         libldap_r-2.3.so.0 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libldap_r-2.3.so.0 (0x00002b10ef0de000)

         /lib64/ld-linux-x86-64.so.2 (0x0000003f8bc00000)

         libdl.so.2 => /lib64/libdl.so.2 (0x00002b10ef332000)

         libkrb5.so.3 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libkrb5.so.3 (0x00002b10ef537000)

         libk5crypto.so.3 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libk5crypto.so.3 (0x00002b10ef7c1000)

         libcom_err.so.3 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libcom_err.so.3 (0x00002b10ef9e6000)

         libkrb5support.so.0 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/libkrb5support.so.0 (0x00002b10efbeb000)

         libresolv.so.2 => /lib64/libresolv.so.2 (0x00002b10efdf2000)

         libfreebl3.so => /lib64/libfreebl3.so (0x00002b10f000d000)

         liblber-2.3.so.0 => /usr/local/greenplum-connectivity-4.3.8.2-build-1/drivers/odbc/psqlodbc-08.02.0500/datadirect-52_64/liblber-2.3.so.0 (0x00002b10f0210000)