impala高级设置之BROADCAST_BYTES_LIMIT

官网地址

https://impala.apache.org/docs/build/html/topics/impala_broadcast_bytes_limit.html

Sets the limit for the size of the broadcast input based on estimated size.

The Impala planner may in rare cases make a bad choice to broadcast a large table or intermediate result and encounter performance problems due to high memory pressure. Setting this limit will make the planner pick a partition based hash join instead of broadcast and avoid such performance problems.

Type: integer

Default:

The default value is 34359738368 (32 GB). A value of 0 causes the option to be ignored.

Examples:

-- Change the limit to 16GB.
set broadcast_bytes_limit=17179869184;

-- Disable the BROADCAST_BYTES_LIMIT.
set broadcast_bytes_limit=0;

这个没啥说的就是广播表的时候 限制其大小 。

这个设置有个什么好处。前面有话里说了。在极少数的情况下 impala会做出一个错误的选择，也就是我们各种join的时候，他会把一个巨大的中间表或者一个大表去广播出去，这个时候一广播用的内存就多了！！！很容易就挂了。

所以说呢 设置这个值小点可以避免这个中问题的发生的，但是小了，就会影响一些正常的查询。

可惜的是我在set里还是没找到该参数。

In Impala 2.0 and later, you can set query options directly through the JDBC and ODBC interfaces by using the SET statement. Formerly, SET was only available as a command within the impala-shell interpreter.

In Impala 2.11 and later, you can set query options for an impala-shell session by specifying one or more command-line arguments of the form --query_option=option=value. See impala-shell Configuration Options for details.

但是提到了这两句话 可能能在jdbc连接串里添加该参数

[devuser@worker03 ~]$ impala-shell -i test-flimpala.woa.com:25003 -d default -k -Q broadcast_bytes_limit=17179869184
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
Opened TCP connection to test-flimpala.woa.com:25003
Connected to test-flimpala.woa.com:25003
Server version: impalad version 3.2.0-cdh6.3.2 RELEASE (build 1bb9836227301b839a32c6bc230e35439d5984ac)
BROADCAST_BYTES_LIMIT is not supported for the impalad being connected to, ignoring.
Query: use `default`
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v3.2.0-cdh6.3.2 (1bb9836) built on Fri Nov  8 07:22:06 PST 2019)

Press TAB twice to see a list of available commands.
***********************************************************************************
Query: use `default`
[test-flimpala.woa.com:25003] default> set
 

 无果。他说通过set可以看到哪些值可以set 结果有没有。全当知识面补充了