Impala重点问题列表
Impala重点问题列表（截止到1.01）
从目前的情况来看，impala的稳定性和易用性还有待提高，在生产环境中部署还存在比较大的风险。
未解决：
· impala-shell命令不兼容Python 2.4
Impala 1.0.1中的impala-shell命令与Python 2.4不兼容, 后者是Red Hat 5的默认Python.
Impala 1.0中的impala-shell命令的-o选项与Python 2.4不兼容。
严重性: 高
解决方案: 在未来版本中修正
变通方法: 使用集成Python2.4的linux, 使用Impala 1.0中的impala-shell包,同时避免使用-o选项。
· 大多数查询有10-20%的性能退化
问题是针对系统并发查询和单一查询之间的性能权衡造成的。目前，单一查询性能比之前版本低，并发查询比之前版本的性能高。
严重性: 高
解决方案: 在未来版本中修正
· Impala不支持使用federated namespaces的集群
Impala不支持在federatednamespaces的集群上运行。impalad进程在运行有基于theorg.apache.hadoop.fs.viewfs.ViewFs类的文件系统的节点上无法启动。
严重性: 待定
变通方法: 在所有Impala节点上使用标准HDFS。
· 表在FROM语句中的顺序对性能优化有非常大的影响
Impala目前没有优化查询中的join顺序，而是按照FROM语句中的书写顺序进行join操作。如果在查询中join右边包含一个多多个大表将导致运行缓慢或由于OOM而造成的Impala崩溃。 例如:
SELECT ... FROM small_table JOIN large_table
严重性: 中
解决方案: 在未来版本中修正
变通方法:修改查询语句,如果可能的话，先join最大的表。例如:
SELECT ... FROM small_table JOIN large_table
应修改为:
SELECT ... FROM large_table JOIN small_table
已解决（0.3之后）：
1.0.1
· Impala parquet扫描器无法读取由其他框架生成的数据文件
· HBase查询遗漏了最后一个region的数据
· Hbase region变化没有被正确地处理
· 某些操作没有释放JNI-allocatedbyte-strings，将导致内存泄露
· 分布式外连接返回结果有误
1.0 GA
· HBase扫描非常慢（由于Impala节点占用了过多了内存）
· 某些查询使用了过多的内存
· HdfsScanNode在 UpdateCounters时崩溃
· 某些在包含大量数据的Parquet表上的聚集查询性能较差
· impala在创建表时没有正确地生成hive元数据
· Constant SELECT语句在子查询中无法工作
· Parquet扫描器在某些查询中将挂起
0.7
· 不支持DDL语句 (CREATE/ALTER/DROP TABLE)
· 不支持Avro
· 不支持对单一查询进行内存限制，所有join右边的表都需要被加载到内存，可能导致Impala由于OOM而崩溃
· 对子查询的结果进行聚集操作将返回错误的结果，如果该子查询包含limit并且在多个节点上进行分布式处理
0.6
· 使用Thrift接口连接Hive元数据时并发查询可能失败
0.5
· Impala在包含大量partitions的表示查询性能较差
· 并发查询可能失败，错误信息为: "Tableobject has not been been initialised : `PARTITIONS`"
0.4
· 包含大limit的查询将被挂起
附录：Impala所有问题列表
Known Issues in the Current Release
The impala-shell command in Impala 1.0.1 doesnot work with Python 2.4, which is the default on Red Hat 5.
For the impala-shell commandin Impala 1.0, the -o option (pipe output to a file) does not workwith Python 2.4.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-396
Severity: High
Workaround: On Linux systems with Python 2.4, use the impala-shell packagefrom Impala 1.0, and avoid using the -o option.
Resolution: To be fixed in a future release
10-20% perfregression for most queries across all table formats
This issue is dueto a performance tradeoff between systems running many queries concurrently,and systems running a single query. Systems running only a single query couldexperience lower performance than in early beta releases. Systems running manyqueries simultaneously should experience higher performance than in the betareleases.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-335
Severity: High
Resolution: To be fixed in a future release
Impala does notsupport running on clusters with federated namespaces
Impala does notsupport running on clusters with federated namespaces. The impalad processwill not start on a node running such a filesystem based on theorg.apache.hadoop.fs.viewfs.ViewFs class.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-77
Severity: Undetermined
AnticipatedResolution: Limitation
Workaround: Use standard HDFS on all Impala nodes.
Order of tablereferences in FROM clause is critical for optimal performance
Impala does notcurrently optimize the join order of queries; instead, it joins tables in theorder in which they are listed in the FROM clause. Queries that contain one ormore large tables on the right hand side of joins (either an explicit joinexpressed as a JOIN statement or a join implicit in the list of tablereferences in the FROM clause) may run slowly or crash Impala due toout-of-memory errors. For example:
SELECT ... FROM small_table JOIN large_table
Severity: Medium
AnticipatedResolution: To be fixed in a future release
Workaround: Modify query, if possible, to join the largest tablefirst. For example:
SELECT ... FROM small_table JOIN large_table
should be modifiedto:
SELECT ... FROM large_table JOIN small_table
Impala INSERTOVERWRITE ... SELECT behavior differs from Hive in that partitions are onlydeleted/re-written if the SELECT statement returns data.
Impala INSERTOVERWRITE ... SELECT behavior differs from Hive in that the partitions are onlydeleted or rewritten if the SELECT statement returns data. Hive always deletesthe data.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-89
Severity: Medium
Workaround: None
Deviation fromHive behavior: Out of range values float/double values are returned as maximumallowed value of type (Hive returns NULL)
Impala behaviordiffers from Hive with respect to out of range float/double values. Out ofrange values are returned as maximum allowed value of type (Hive returns NULL).
Severity: Low
Workaround: None
Deviation fromHive behavior: Impala does not do implicit casts between string and numeric andboolean types.
Severity: Low
AnticipatedResolution: None
Workaround: Use explicit casts.
If Hue and Impalaare installed on the same host, and if you configure Hue Beeswax in CDH 4.1 toexecute Impala queries, Beeswax cannot list Hive tables and shows an error onBeeswax startup.
Hue requiresBeeswaxd to be running in order to list the Hive tables. Because of a portconflict bug in Hue in CDH4.1 when Hue and Impala are installed on the samehost, an error page is displayed when you start the Beeswax application, andwhen you open the Tables pagein Beeswax.
Severity: High
AnticipatedResolution: Fixed in an upcomingCDH4 release
Workarounds: Choose one of the following workarounds (but onlyone):
· Install Hue andImpala on different hosts. OR
· Upgrade toCDH4.1.2 and add the following property in the beeswax section ofthe /etc/hue/hue.ini configuration file:
beeswax_meta_server_only=9004
OR
· If you are usingCDH4.1.1 and you want to install Hue and Impala on the same host, change thecode in this file:
/usr/share/hue/apps/beeswax/src/beeswax/management/commands/beeswax_server.py
Replace line 66:
str(beeswax.conf.BEESWAX_SERVER_PORT.get()),
With this line:
'8004',
Beeswaxd will then use port 8004.
Note:
If you used Cloudera Manager to install Impala, refer tothe Cloudera Manager release notes for information about using an equivalentworkaround by specifying the beeswax_meta_server_only=9004 configurationvalue in the Hue Service Configuration SafetyValve.
Known Issues Fixed in the 1.0.1 Release
This section liststhe most significant issues fixed in Impala 1.0.1. For the full list of fixedissues, see this report in the JIRA system.
Impala parquetscanner can not read all data files generated by other frameworks
Impala might issuean erroneous error message when processing a Parquet data file produced by anon-Impala Hadoop component.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-333
Severity: High
Resolution: Fixed
Impala is unableto query RCFile tables which describe fewer columns than the file's header.
If an RCFile tabledefinition had fewer columns than the fields actually in the data files,queries would fail.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-293
Severity: High
Resolution: Fixed
Impala does notcorrectly substitute _HOST with hostname in --principal
The _HOST placeholderin the --principal startup option was not substituted with the correcthostname, potentially leading to a startup error in setups using Kerberosauthentication.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-351
Severity: High
Resolution: Fixed
HBase query missedthe last region
A query for anHBase table could omit data from the last region.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-
Severity: High
Resolution: Fixed
Hbase regionchanges are not handled correctly
After a region inan HBase table was split or moved, an Impala query might return incomplete orout-of-date results.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-300
Severity: High
Resolution: Fixed
Query state forsuccessful create table is EXCEPTION
After asuccessful CREATE TABLE statement, the corresponding query state would beincorrectly reported as EXCEPTION.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-349
Severity: High
Resolution: Fixed
Double checkrelease of JNI-allocated byte-strings
Operationsinvolving calls to the Java JNI subsystem (for example, queries on HBasetables) could allocate memory but not release it.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-358
Severity: High
Resolution: Fixed
Impala returns 0for bad time values in UNIX_TIMESTAMP, Hive returns NULL
Impala returns 0for bad time values in UNIX_TIMESTAMP, Hive returns NULL.
Impala:
impala> select UNIX_TIMESTAMP('10:02:01') ;
impala> 0
Hive:
hive> select UNIX_TIMESTAMP('10:02:01') FROM tmp;
hive> NULL
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-16
Severity: Low
AnticipatedResolution: Fixed
INSERT INTO TABLESELECT
Insert INTO TABLESELECT
Severity: Low
AnticipatedResolution: Fixed
Distributed outerjoin returns wrong result
When you executean outer join query in distributed mode, Impala will return more rows thanexpected (that is, the wrong result).
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-90
Severity: Medium
AnticipatedResolution: To be fixed in a futurerelease
Workaround: None
Known Issues Fixed in the 1.0 GA Release
Here are the majoruser-visible issues fixed in Impala 1.0. For a full list of fixed issues,see this report in the public issue tracker.
Impala does not properly modify DATETIMEand TIMESTAMP partition keys via DML or DDL statements
Impala does notproperly modify DATETIME and TIMESTAMP partition keys via DML or DDLstatements. Modifying partition columns of this datatype may result in failedqueries and/or table metadata loading problems.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-238
Severity: High
Resolution: Fixed
Undeterministicallyreceive "ERROR: unknown row bach destination..." and "ERROR:Invalid query handle" from impala shell when running union query
A query containingboth UNION and LIMIT clauses could intermittently cause the impalad process tohalt with a segmentation fault.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-183
Severity: High
Resolution: Fixed
Insert with NULLpartition keys results in SIGSEGV.
An INSERT statementspecifying a NULL value for one of the partitioning columns couldcause the impalad process to halt with a segmentation fault.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-190
Severity: High
Resolution: Fixed
INSERT queriesdon't show completed profiles on the debug webpage
In the Impala webuser interface, the profile page for an INSERT statementshowed obsolete information for the statement once it was complete.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-217
Severity: High
Resolution: Fixed
Impala HBase scanis very slow
Queries involvingan HBase table could be slower than expected, due to excessive memory usage onthe Impala nodes.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-231
Severity: High
Resolution: Fixed
Add some libraryversion validation logic to impalad when loading impala-lzo shared library
No validation wasdone to check that the impala-lzo shared library was compatible with the version ofImpala, possibly leading to a crash when using LZO-compressed text files.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-234
Severity: High
Resolution: Fixed
Workaround: Always upgrade the impala-lzo library atthe same time as you upgrade Impala itself.
Problems insertinginto tables with TIMESTAMP partition columns leading table metadata loadingfailures and failed dchecks
INSERT statements for tables partitioned on columnsinvolving datetime types could appear to succeed, but cause errors forsubsequent queries on those tables. The problem was especially serious if animproperly formatted timestamp value was specified for the partition key.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-238
Severity: Critical
Resolution: Fixed
Ctrl-C sometimesinterrupts shell in system call, rather than cancelling query
Pressing Ctrl-C inthe impala-shell interpreter could sometimes display an error andreturn control to the shell, making it impossible to cancel the query.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-243
Severity: Critical
Resolution: Fixed
Empty stringpartition value causes metastore update failure
Specifying anempty string or NULL for a partition key in an INSERT statementwould fail.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-252
Severity: High
Resolution: Fixed. The behavior for empty partition keys wasmade more compatible with the corresponding Hive behavior.
Round() does notoutput the right precision
The round() function didnot always return the correct number of significant digits.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-266
Severity: High
Resolution: Fixed
Cannot cast stringliteral to string
Casting from astring literal back to the same type would cause an "invalid typecast" error rather than leaving the original value unchanged.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-267
Severity: High
Resolution: Fixed
Excessive memusage for certain queries which are very selective
Some queries that returnedvery few rows experienced unnecessary memory usage.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-288
Severity: High
Resolution: Fixed
HdfsScanNodecrashes in UpdateCounters
A serious errorcould occur for relatively small and inexpensive queries.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-289
Severity: High
Resolution: Fixed
Parquetperformance issues on large dataset
Certainaggregation queries against Parquet tables were inefficient due to lower thanrequired thread utilization.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-292
Severity: High
Resolution: Fixed
impala notpopulating hive metadata correctly for create table
The Impala CREATE TABLE commanddid not fill in the owner and tbl_type columns in the Hive metastoredatabase.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-295
Severity: High
Resolution: Fixed. The metadata was made more Hive-compatible.
impala daemons dieif statestore goes down
The impalad instances ina cluster could halt when the statestored process becameunavailable.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-312
Severity: High
Resolution: Fixed
Constant SELECTclauses do not work in subqueries
A subquery wouldfail if the SELECT statement inside it returned a constant valuerather than querying a table.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-67
Severity: High
Resolution: Fixed
Right outer Joinincludes NULLs as well and hence wrong result count
The result setfrom a right outer join query could include erroneous rows containing NULL values.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-90
Severity: High
Resolution: Fixed
Parquet scanner hangs for some queries
The Parquetscanner non-deterministically hangs when executing some queries.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-204
Severity: Medium
Resolution: Fixed
Known Issues Fixed in Version 0.7 of the Beta Release
Impala does not gracefully handleunsupported Hive table types (INDEX and VIEW tables)
When attempting toload metadata from an unsupported Hive table type (INDEX and VIEW tables),Impala fails with an unclear error message.
Cloudera Bug:https://issues.cloudera.org/browse/IMPALA-167
Severity: Low
Resolution: Fixed in 0.7
DDL statements(CREATE/ALTER/DROP TABLE) are not supported in the Impala Beta Release
Severity: Medium
Resolution: Fixed in 0.7
Avro is notsupported in the Impala Beta Release
Severity: Medium
Resolution: Fixed in 0.7
Workaround: None
Impala does notcurrently allow limiting the memory consumption of a single query
It is currentlynot possible to limit the memory consumption of a single query. All tables onthe right hand side of JOIN statements need to be able to fit in memory. Ifthey do not, Impala may crash due to out of memory errors.
Severity: High
Resolution: Fixed in 0.7
Aggregate of asubquery result set returns wrong results if the subquery contains a 'limit'and data is distributed across multiple nodes
Aggregate of asubquery result set returns wrong results if the subquery contains a 'limit'clause and data is distributed across multiple nodes. From the query plan, itlooks like we are just summing the results from each slave.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-20
Severity: Low
Resolution: Fixed in 0.7
Partition pruningfor arbitrary predicates that are fully bound by a particular partition column
We currently can'tutilize a predicate like "country_code in ('DE', 'FR', 'US')" to dopartitioning pruning, because that requires an equality predicate or a binarycomparison.
We should create asuperclass of planner.ValueRange, ValueSet, that can be constructed with anarbitrary predicate, and whose isInRange(analyzer, valueExpr) constructs aliteral predicate by substitution of the valueExpr into the predicate.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-144
Severity: Medium
Resolution: Fixed in 0.7
Known Issues Fixed in Version 0.6 of the Beta Release
Impala reads theNameNode address and port as command line parameters
Impala reads theNameNode address and port as command line parameters rather than reading themfrom core-site.xml. Updating the NameNode address in the core-site.xml filedoes not propagate to Impala.
Severity: Low
Resolution: Fixed in 0.6 - Impala reads the namenode locationand port from the Hadoop configuration files, though setting -nn and -nn_port overridesthis. Users are advised not to set -nn or -nn_port.
Queries may failon secure environment due to impalad Kerberos ticket expiration
Queries may failon secure environment due to impalad Kerberos tickets expiring.This can happen if the Impala -kerberos_reinit_interval flag is setto a value ten minutes or less. This may lead to an impalad requesting aticket with a lifetime that is less than the time to the next ticket renewal.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-64
Severity: Medium
Resolution: Fixed in 0.6
Concurrent queriesmay fail when Impala uses Thrift to communicate with the Hive Metastore
Concurrent queriesmay fail when Impala is using Thrift to communicate with part of the HiveMetastore such as the Hive Metastore Service. In such a case, the errorget_fields failed: out of sequence response" may occur because Impala shared a single Hive MetastoreClient connection across threads. With Impala 0.6, a separate connection isused for each metadata request.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-48
Severity: Low
Resolution: Fixed in 0.6
impalad fails tostart if unable to connect to the Hive Metastore
Impala fails tostart if it is unable to establish a connection with the Hive Metastore. Thisbehavior was fixed, allowing Impala to start, even when no Metastore isavailable.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-58
Severity: Low
Resolution: Fixed in 0.6
Impala treatsdatabase names as case-sensitive in some contexts
In some queries(including "USE database" statements), database names are treated ascase-sensitive. This may lead queries to fail with an IllegalStateException.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-44
Severity: Medium
Resolution: Fixed in 0.6
Impala does notignore hidden HDFS files
Impala does notignore hidden HDFS files, meaning those files prefixed with a period '.' orunderscore '_'. This diverges from Hive/MapReduce, which skips these files.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-18
Severity: Low
Resolution: Fixed in 0.6
Known Issues Fixed in Version 0.5 of the Beta Release
Impala may havereduced performance on tables that contain a large number of partitions
Impala may havereduced performance on tables that contain a large number of partitions. Thisis due to extra overhead reading/parsing the partition metadata.
Severity: High
Resolution: Fixed in 0.5
Backend clientconnections not getting cached causes an observable latency in secure clusters
Backend impaladsdo not cache connections to the coordinator. On a secure cluster, thisintroduces a latency proportional to the number of backend clients involved inquery execution, as the cost of establishing a secure connection is much higherthan in the non-secure case.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-38
Severity: Medium
Resolution: Fixed in 0.5
Concurrent queriesmay fail with error: "Table object has not been been initialised :`PARTITIONS`"
Concurrent queriesmay fail with error: "Tableobject has not been been initialised : `PARTITIONS`". This was due to a lack of locking in the Impalatable/database metadata cache.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-30
Severity: Medium
Resolution: Fixed in 0.5
UNIX_TIMESTAMPformat behaviour deviates from Hive when format matches a prefix of the timevalue
The ImpalaUNIX_TIMESTAMP(val, format) operation compares the length of format and val andreturns NULL if they do not match. Hive instead effectively truncates val tothe length of the format parameter.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-15
Severity: Medium
Resolution: Fixed in 0.5
Known Issues Fixed in Version 0.4 of the Beta Release
Impala fails torefresh the Hive metastore if a Hive temporary configuration file is removed
Impala is impactedby Hive bug HIVE-3596 which may cause metastore refreshes to fail if a Hive temporaryconfiguration file is deleted (normally located at /tmp/hive-
Severity: Medium
AnticipatedResolution: To be fixed in a futurerelease
Workaround: Restart the impalad service. Usethe impalad log to check for metadata refresh errors.
lpad/rpad builtinfunctions is not correct.
The lpad/rpadbuiltin functions generate the wrong results.
Severity: Mild
Resolution: Fixed in 0.4
Files with .gzextension reported as 'not supported'
Compressed fileswith extensions incorrectly generate an exception.
ClouderaBug: https://issues.cloudera.org/browse/IMPALA-14
Severity: High
Resolution: Fixed in 0.4
Queries with largelimits would hang.
Some queries withlarge limits were hanging.
Severity: High
Resolution: Fixed in 0.4
Order by on astring column produces incorrect results if there are empty strings
Severity: Low
Resolution: Fixed in 0.4
Known Issues Fixed in Version 0.3 of the Beta Release
All table loadingerrors show as unknown table
If Impala isunable to load the metadata for a table for any reason, a subsequent queryreferring to that table will return an unknown table errormessage, even if the table is known.
Severity: Mild
Resolution: Fixed in 0.3
A table thatcannot be loaded will disappear from SHOW TABLES
After failing toload metadata for a table, Impala removes that table from the list of knowntables returned in SHOWTABLES. Subsequent attempts to querythe table returns 'unknown table', even if the metadata for that table isfixed.
Severity: Mild
Resolution: Fixed in 0.3
Impala cannot readfrom HBase tables that are not created as external tables in the hivemetastore.
Attempting toselect from these tables fails.
Severity: Medium
Resolution: Fixed in 0.3
Certain queriesthat contain OUTER JOINs may return incorrect results
Queries thatcontain OUTER JOINs may not return the correct results if there are predicatesreferencing any of the joined tables in the WHERE clause.
Severity: Medium
Resolution: Fixed in 0.3.
Known Issues Fixed in Version 0.2 of the Beta Release
Subqueries whichcontain aggregates cannot be joined with other tables or Impala may crash
Subqueries whichcontain an aggregate cannot be joined with another table or Impala may crash.For example: SELECT * FROM (SELECT sum(col1) FROM some_table GROUP BY col1) t1JOIN other_table ON (...).
Severity: Medium
Resolution: Fixed in 0.2
An insert with alimit that runs as more than one query fragment inserts more rows than thelimit.
For example:
INSERT OVERWRITE TABLE test SELECT * FROM test2 LIMIT 1;
Severity: Medium
Resolution: Fixed in 0.2
Query with limitclause might fail.
For example:
SELECT * FROM test2 LIMIT 1;
Severity: Medium
Resolution: Fixed in 0.2
Files inunsupported compression formats are read as plain text.
Attempting to readsuch files does not generate a diagnostic.
Severity: Medium
Resolution: Fixed in 0.2
Impala serverraises a null pointer exception when running an HBase query.
When querying anHBase table whose row-key is string type, the Impala server may raise a nullpointer exception.
Severity: Medium
Resolution: Fixed in 0.2