Hp Vertica数据库Perl编程

Hp Vertica数据库Perl编程

Perl语言有Database Interface module,叫做DBI。

DBI执行特定的数据库的任务，依赖Database Driver modules (DBDs)

DBI：和数据库的交互，具有持续性。

Perl操控HP Vertica方式：

DBI模块的DBD::ODBC和HP Vertica's ODBC driver交互

See the CPAN pages for Perl's DBI http://search.cpan.org/dist/DBI/DBI.pm  and DBD::ODBC http://search.cpan.org/~mjevans/DBD-ODBC-1.43/ODBC.pm

 

Perl Client编程先决条件：

【1】The HP Vertica ODBC drivers安装和配置

注： Installing the HP Vertica Client Drivers 1.Client Driver标准： ODBC drivers符合 ODBC 3.5.1标准； JDBC drivers 符合 JDK5标准 ADO.NET drivers符合framework 3.0标准 2.windows客户端下载32-bit or 64-bit client installer 3.java应用，要加载vertica_jdk_5.jar，例如 java -classpath /opt/vertica/java/lib/vertica_jdk_5.jar myapplication.class

【2】Windows安装Perl

ActivePerl http://www.activestate.com/activeperl/ Strawberry Perl http://strawberryperl.com/

【3】Perl Driver Modules (DBI 和DBD::ODBC)

HP Vertica支持的perl module

DBI version 1.609 (DBI-1.609.tar.gz) DBD::ODBC version 1.22 (DBD-ODBC-1.22.tar.gz)

测试DBI是否安装

C:\Windows\System32>perl -e "use DBI;" C:\Windows\System32>

测试DBD::ODBC是否安装

C:\Windows\System32>perl -e "use DBD::ODBC;" C:\Windows\System32>

或者用脚本来同时验证

#!/usr/bin/perl use strict; # Attempt to load the DBI module in an eval using require. Prevents # script from erroring out if DBI is not installed. eval { require DBI; DBI->import(); }; if ($@) { # The eval failed, so DBI must not be installed print "DBI module is not installed\n"; } else { # Eval was successful, so DBI is installed print "DBI Module is installed\n"; # List the drivers that DBI knows about. my @drivers = DBI->available_drivers; print "Available Drivers: \n"; foreach my $driver (@drivers) { print "\t$driver\n"; } # See if DBD::ODBC is installed by searching driver array. if (grep {/ODBC/i} @drivers) { print "\nDBD::ODBC is installed.\n"; # List the ODBC data sources (DSNs) defined on the system print "Defined ODBC Data Sources:\n"; my @dsns = DBI->data_sources('ODBC'); foreach my $dsn (@dsns) { print "\t$dsn\n"; } } else { print "DBD::ODBC is not installed\n"; } }

输出结果为

DBI Module is installed Available Drivers: CSV DBM ExampleP File Gofer ODBC Oracle Pg Proxy SQLite Sponge DBD::ODBC is installed. Defined ODBC Data Sources: dbi:ODBC:dBASE Files dbi:ODBC:Excel Files dbi:ODBC:MS Access Database

【4】安装缺少的Perl Modules

缺少dbi:ODBC:VerticaDSN

对于ActiveState Perl for Windows, 使用Perl Package Manager (PPM)

使用Perl连接HP Vertica

使用Perl的DBI模块的connect函数连接vertica数据库， connect函数需要数据源字符串参数和用户名，密码，连接属性等。数据源字符串以“dbi:ODBC”开头，告诉DBI模块使用DBD::ODBC driver来连接数据库，余下的数据源字符串，就由DBD::ODBC driver来解释了。经常使用DSN来连接数据库，例如dbi:ODBC:VerticaDSN。

如果没有连接成功数据库，通过$DBI::errstr查看报错信息，connect函数返回undef。

注：Perl DBI Connection Attributes AutoCommit to false：  # Create a hash that holds attributes for the connection my $attr = {AutoCommit => 0}; # Open a connection using a DSN. Supply the username and password. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN","ExampleUser","password123", $attr); print "The AutoCommit attribute is: " . $dbh->{AutoCommit} . "\n";

 

【1】 通过DNS连接数据库

On Linux and other UNIX-like operating systems, the HP Vertica ODBC driver's name is assigned in the system's odbcinst.ini file. For example, if your /etc/odbcint.ini contains the following: [HPVertica] Description = HP Vertica ODBC Driver Driver = /opt/vertica/lib64/libverticaodbc.so

【2】 实例1通过DNS连接HP vertica

#!/usr/bin/perl use strict; use DBI; # Create a hash that holds attributes for the connection my $attr = { RaiseError => 1, # Make database errors fatal to script AutoCommit => 0, # Prevent statements from committing # their transactions. }; # Open a connection using a DSN. Supply the username and password. my $dbh = DBI->connect("dbi:ODBC: ansi_local",$attr); if (defined $dbh->err) { # Connection failed. die "Failed to connect: $DBI::errstr"; } print "Connected!\n";   # The database handle lets you access the connection attributes directly: print "The AutoCommit attribute is: " . $dbh->{AutoCommit} . "\n"; print "The RaiseError attribute is: " . $dbh->{RaiseError} . "\n"; # And you can change values, too... $dbh->{AutoCommit} = 1; print "The AutoCommit attribute is now: " . $dbh->{AutoCommit} . "\n"; $dbh->disconnect();

【3】 同事Perl变量来定义数据源字符串连接数据

 

#!/usr/bin/perl use strict; use DBI; my $server='VerticaHost'; my $port = '5433'; my $database = 'VMart'; my $user = 'ExampleUser'; my $password = 'password123'; # Connect without a DSN by supplying all of the information for the connection. # The DRIVER value on UNIX platforms depends on the entry in the odbcinst.ini file. my $dbh = DBI->connect("dbi:ODBC:DRIVER={Vertica};Server=$server;" .                        "Port=$port;Database=$database;UID=$user;PWD=$password")                         or die "Could not connect to database: " . DBI::errstr; print "Connected!\n"; $dbh->disconnect();

使用Perl执行SQL语句

例如

#!/usr/bin/perl use strict; use DBI; # Disable autocommit my $attr = {AutoCommit => 0}; # Open a connection using a DSN. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN","ExampleUser","password123", $attr); unless (defined $dbh) { # Conection failed. die "Failed to connect: $DBI::errstr"; }   # You can use the do function to perform DDL commands. # Drop any existing table. $dbh->do("DROP TABLE IF EXISTS TEST CASCADE;");   # Create a table to hold data. $dbh->do("CREATE TABLE TEST( \  C_ID INT, \  C_FP FLOAT,\  C_VARCHAR VARCHAR(100),\  C_DATE DATE, C_TIME TIME,\  C_TS TIMESTAMP,\  C_BOOL BOOL)");   # Commit changes and exit. $dbh->commit(); $dbh->disconnect();

使用Perl批量加载数据

1. 批量加载需要设置DBI的 AutoCommit connection属性false

2. 使用prepare函数，包括插入values值占位符的insert语句

# Prepare an INSERT statement for the test table $sth = $dbh->prepare("INSERT into test values(?,?,?,?,?,?,?)");

3. 给占位符分配数据；比如，每个占位符要插入的值构建一个矩阵。

4. 调用insert语句句柄的execute函数来插入一行数据；

5. 重复3,4步骤

6. 调用数据库句柄的commit函数，提交插入的数据；

例如

#!/usr/bin/perl use strict; use DBI; # Create a hash reference that holds a hash of parameters for the # connection my $attr = {AutoCommit => 0, # Turn off autocommit           PrintError => 0 # Turn off automatic error printing. # This is handled manually.            };             # Open a connection using a DSN. Supply the username and password. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN","ExampleUser","password123", $attr);   if (defined DBI::err) { # Conection failed. die "Failed to connect: $DBI::errstr"; } print "Connection AutoCommit state is: " . $dbh->{AutoCommit} . "\n"; # Create table to hold inserted data $dbh->do("DROP TABLE IF EXISTS TEST CASCADE;") or die "Could not drop table"; $dbh->do("CREATE TABLE TEST( \ C_ID INT, \ C_FP FLOAT,\ C_VARCHAR VARCHAR(100),\ C_DATE DATE, C_TIME TIME,\ C_TS TIMESTAMP,\ C_BOOL BOOL)") or die "Could not create table";   # Populate an array of arrays with values. One of these rows contains # data that will not be sucessfully inserted. Another contains an # undef value, which gets inserted into the database as a NULL. my @data =( [1,1.111,'Hello World!','2001-01-01','01:01:01' ,'2001-01-01 01:01:01','t'], [2,2.22222,'How are you?','2002-02-02','02:02:02' ,'2002-02-02 02:02:02','f'], ['bad value',2.22222,'How are you?','2002-02-02','02:02:02' ,'2002-02-02 02:02:02','f'], [4,4.22222,undef,'2002-02-02','02:02:02' ,'2002-02-02 02:02:02','f'], );   # Create a prepared statement to use parameters for inserting values. my $sth = $dbh->prepare_cached("INSERT into test values(?,?,?,?,?,?,?)"); my $rowcount = 0; # Count # of rows   # Loop through the arrays to insert values foreach my $tuple (@data) { $rowcount++; # Insert the row my $retval = $sth->execute(@$tuple); # See if the row was successfully inserted. if ($retval == 1) { # Value of 1 means the row was inserted (1 row was affected by insert) print "Row $rowcount successfully inserted\n"; } else { print "Inserting row $rowcount failed"; # Error message is not set on some platforms/versions of DBUI. Check to # ensure a message exists to avoid getting an unitialized var warning. if ($sth->err()) {  print ": " . $sth->errstr();  } print "\n"; } }   #you need to use commit for batched # data to actually be committed into the database. $dbh->commit(); $dbh->disconnect();

 

使用COPY LOCAL语句加载文件数据

例如，数据文件data.txt存放：

1|Georgia|Gomez|[email protected]|1937-10-03

2|Abdul|Alexander|[email protected]|1941-03-10

use strict; use DBI; # Filesystem path handling module use File::Spec; # Create a hash reference that holds a hash of parameters for the # connection. my $attr = {AutoCommit => 0}; # Turn off AutoCommit # Open a connection using a DSN. Supply the username and password. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN","ExampleUser","password123", $attr) or die "Failed to connect: $DBI::errstr"; print "Connected!\n";   # Drop any existing table. $dbh->do("DROP TABLE IF EXISTS Customers CASCADE;"); # Create a table to hold data. $dbh->do("CREATE TABLE Customers( \ ID INT, \ FirstName VARCHAR(100),\ LastName VARCHAR(100),\ Email VARCHAR(100),\ Birthday DATE)");   # Find the absolute path to the data file located in the current working # directory and named data.txt my $currDir = File::Spec->rel2abs(File::Spec->curdir()); my $dataFile = File::Spec->catfile($currDir, 'data.txt'); print "Loading file $dataFile\n"; # Load local file using copy local. Return value is the # of rows affected # which equates to the number of rows inserted. my $rows = $dbh->do("COPY Customers FROM LOCAL '$dataFile' DIRECT") or die $dbh->errstr; print "Copied $rows rows into database.\n"; $dbh->commit();   # Prepare a query to get the first 15 rows of the results my $sth = $dbh->prepare("SELECT * FROM Customers WHERE ID < 15 \ ORDER BY ID"); $sth->execute() or die "Error querying table: " . $dbh->errstr; my @row; # Pre-declare variable to hold result row used in format statement.   # Use Perl formats to pretty print the output. Declare the heading for the # form. format STDOUT_TOP = ID First Last EMail Birthday == ===== ==== ===== ======== # The Perl write statement will output a formatted line with values from the # @row array. See http://perldoc.perl.org/perlform.html for details.   format STDOUT = @> @<<<<<<<<<<<<< @<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<<<<<<<<< @<<<<<<<<< @row .   # Loop through result rows while we have them while (@row = $sth->fetchrow_array()) { write; # Format command does the work of extracting the columns from # the @row array and writing them out to STDOUT. } # Call commit to prevent Perl from complaining about uncommitted transactions # when disconnecting $dbh->commit(); $dbh->disconnect();

 

 

使用Perl进行数据查询

1. 使用prepare函数创建查询语句；

2. 执行查询，execute函数

3. 通过某一个函数检索加工的结果

（1） fetchrow_array获取一行数据；

（2） fetchall_array获取一个数组，包含全部的result set(数据量较大，不适合使用)

#!/usr/bin/perl use strict; use DBI; my $attr = {RaiseError => 1 }; # Make errors fatal to the Perl script. # Open a connection using a DSN. Supply the username and password. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN","ExampleUser","password123", $attr);   # Prepare a query to get the content of the table my $sth = $dbh->prepare("SELECT * FROM TEST ORDER BY C_ID ASC"); # Execute the query by calling execute on the statement handle $sth->execute();   # Loop through result rows while we have them, getting each row as an array while (my @row = $sth->fetchrow_array()) { # The @row array contains the column values for this row of data # Loop through the column values foreach my $column (@row) { if (!defined $column) { # NULLs are signaled by undefs. Set to NULL for clarity $column = "NULL"; } print "$column\t"; # Output the column separated by a tab } print "\n"; }   $dbh->disconnect();

【1】 Binding Variables to Column Values

语句的句柄有bind_columns函数。使用变量来接受查询结果，比数组来存放结果，更方便。

#!/usr/bin/perl use strict; use DBI; my $attr = {RaiseError => 1 }; # Make SQL errors fatal to the Perl script. # Open a connection using a DSN. Supply the username and password. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN32","ExampleUser","password123", $attr); # Prepare a query to get the content of the table my $sth = $dbh->prepare("SELECT * FROM TEST ORDER BY C_ID ASC"); $sth->execute();   # Create a set of variables to bind to the column values. my ($C_ID, $C_FP, $C_VARCHAR, $C_DATE, $C_TIME, $C_TS, $C_BOOL); # Bind the variable references to the columns in the result set. $sth->bind_columns(\$C_ID, \$C_FP, \$C_VARCHAR, \$C_DATE, \$C_TIME, \$C_TS, \$C_BOOL);   # Now, calling fetch() to get a row of data updates the values of the bound # variables. Continue calling fetch until it returns undefined. while ($sth->fetch()) { # Note, you should always check that values are defined before using them, # since NULL values are translated into Perl as undefined. For this # example, just check the VARCHAR column for undefined values. if (!defined $C_VARCHAR) { $C_VARCHAR = "NULL"; } # Just print values separated by tabs. print "$C_ID\t$C_FP\t$C_VARCHAR\t$C_DATE\t$C_TIME\t$C_TS\t$C_BOOL\n"; } $dbh->disconnect();

【2】 使用selectrow_array只返回一个结果

比如show local;

en_US@collation=binary (LEN_KBINARY)

#!/usr/bin/perl use strict; use DBI; my $attr = {RaiseError => 1 }; # Make SQL errors fatal to the Perl script. # Open a connection using a DSN. Supply the username and password. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN","ExampleUser","password123", $attr);   # Demonstrate setting/getting locale. # Use selectrow_array to combine preparing a statement, executing it, and # getting an array as a result. my @localerv = $dbh->selectrow_array("SHOW LOCALE;"); # The locale name is the 2nd column (array index 1) in the result set. print "Locale: $localerv[1]\n";   # Use do() to execute a SQL statement to set the locale. $dbh->do("SET LOCALE TO en_GB"); # Get the locale again.   @localerv = $dbh->selectrow_array("SHOW LOCALE;"); print "Locale is now: $localerv[1]\n"; $dbh->disconnect();

Perl 和 HP Vertica的数据类型转换

Perl是一种松散类型编程语言不分配特定的数据类型值。它转换字符串和数值之间基于值上执行的操作。

1. DBI将Vertica NULL值转化为Perl的定义(undef)值，因此（When reading data from columns that can contain NULL values, you should always test whether a value is defined before using it.）

2. 默认，DBI假设列的类型是varchar；DBI会完成字符串和数值的转换；

3. 浮点数没有小数点的时候，DBI会将浮点数转换为整形；

Perl 支持Unicode

Perlunicode http://perldoc.perl.org/perlunicode.html perlunitut  http://perldoc.perl.org/perlunitut.html

DBD::ODBC需要兼容unicode的支持

http://search.cpan.org/~mjevans/DBD-ODBC-1.43/ODBC.pm

连接有个熟悉odbc_has_unicode

测试的程序

#!/usr/bin/perl use strict; use DBI; # Open a connection using a DSN. my $dbh = DBI->connect("dbi:ODBC:VerticaDSN","ExampleUser","password123"); unless (defined $dbh) { # Conection failed. die "Failed to connect: $DBI::errstr"; } # Output to a file. Displaying Unicode characters to a console or terminal # window has many problems. This outputs a UTF-8 text file that can # be handled by many Unicode-aware text editors: open OUTFILE, '>:utf8', "unicodeout.txt"; # See if the DBD::ODBC driver was compiled with Unicode support. If this returns # 1, your Perl script will get get strings from the driver with the UTF-8 # flag set on them, ensuring that Perl handles them correctly. print OUTFILE "Was DBD::ODBC compiled with Unicode support? " . $dbh->{odbc_has_unicode} . "\n"; # Create a table to hold VARCHARs $dbh->do("DROP TABLE IF EXISTS TEST CASCADE;"); # Create a table to hold data. Remember that the width of the VARCHAR column # is the number of bytes set aside to store strings, which often does not equal # the number of characters it can hold when it comes to Unicode! $dbh->do("CREATE TABLE test( C_VARCHAR VARCHAR(100) )"); print OUTFILE "Inserting data...\n"; # Use Do to perform simple inserts $dbh->do("INSERT INTO test VALUES('Hello')"); # This string contains several non-latin accented characters and symbols, encoded # with Unicode escape notation. They are converted by Perl into UTF-8 characters $dbh->do("INSERT INTO test VALUES('My favorite band is " . "\N{U+00DC}ml\N{U+00E4}\N{U+00FC}t \N{U+00D6}v\N{U+00EB}rk\N{U+00EF}ll" . " \N{U+263A}')"); # Some Chinese (Simplified) characters. This again uses escape sequence # that Perl translates into UTF-8 characters. $dbh->do("INSERT INTO test VALUES('\x{4F60}\x{597D}')"); print OUTFILE "Getting data...\n"; # Prepare a query to get the content of the table my $sth = $dbh->prepare_cached("SELECT * FROM test"); # Execute the query by calling execute on the statement handle $sth->execute(); # Loop through result rows while we have them while (my @row = $sth->fetchrow_array()) { # Loop through the column values foreach my $column (@row) { print OUTFILE "$column\t"; } print OUTFILE "\n"; } close OUTFILE; $dbh->disconnect();

结果

Was DBD::ODBC compiled with Unicode support? 1 Inserting data... Getting data... Hello My favorite band is ?ml?üt ?v?rk?ll ? 你好

 

************************************************************** ** 欢迎转发，注明原文：blog.csdn.net/clark_xu   徐长亮的专栏 ** 谢谢您的支持，欢迎关注微信公众号：clark_blog  **************************************************************