Kudu异常总结

面试：你懂什么是分布式系统吗？Redis分布式锁都不会？>>>   

1、impala不支持kudu表的关键字：PARTITIONED - LOCATION - ROWFORMAT（官方链接：Impala Keywords Not Supported for Kudu Tables），例如执行如下语句会报错：

CREATE TABLE my_first_table2
(
  id BIGINT,
  name STRING,
  PRIMARY KEY(id)
)
PARTITION BY HASH PARTITIONS 16
STORED AS KUDU
LOCATION 'hdfs://namenodetest01.bi/user/impala';

异常信息：

ERROR: AnalysisException: LOCATION cannot be specified for a Kudu table.

2、kudu表的主键必须在建表语句中列的最前面，并且不为空，例如执行如下语句会报错：

CREATE TABLE `t_scd_order_kudu_2` (
  `id` bigint,
  `order_no` string,
  `member_id` bigint,
  `driver_id` bigint,
  `vehicle_id` bigint,
  `origin` smallint,
  `start_city_id` int,
  `end_city_id` int,
  `estimate_board_time` string,
  `actual_board_time` string,
  `estimate_car_type` int,
  `actual_car_type` int,
  `service_type_id` smallint,
  `status` smallint,
  `balance_status` smallint,
  `balance_time` string,
  `passenger` string,
  `passenger_mobile` string,
  `comment_status` tinyint,
  `remark` string,
  `create_emp` bigint,
  `create_time` string,
  `modify_emp` bigint,
  `modify_time` string,
  `ts` string,
  `dispatch_status` smallint,
  `device_no` string,
  `ent_id` bigint,
  `invalid_type` smallint,
  `dispatch_flag` smallint,
  `share_status` smallint,
  `last_dispatch_flag` smallint,
  `home_popup` tinyint,
  `detail_popup` tinyint,
  `red_point` tinyint,
  `free_ride` tinyint,
  `ent_dept_id` bigint,
  `ent_type_id` int,
  `member_name` string,
  `member_mobile` string,
  `driver_name` string,
  `driver_mobile` string,
  `vehicle_no` string,
  `car_type_name` string,
  `ent_name` string,
  `ent_dept_name` string,
  `ent_type_name` string,
  `ent_pay_type` tinyint,
  `ent_code` string,
  `effect_response_rate` tinyint,
  `confirm_board_time` string,
  `invoice_status` smallint,
  `init_estimate_board_time` string,
  `passenger_country_code` smallint,
  `member_country_code` smallint,
  `delay_dispatch_time` int,
  `virtual_member_mobile` string,
  `virtual_passenger_mobile` string,
  `end_pl_city_code` string,
  `pl_city_code` string,
  `estimate_start_city_code` string,
  `estimate_end_city_code` string,
  `pay_mode` tinyint,
  day_of_time string,
  PRIMARY KEY (id, day_of_time)
)PARTITION BY HASH(day_of_time) PARTITIONS 16 STORED AS KUDU;

异常信息：

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::kudu_test.t_scd_order_kudu_2'
CAUSED BY: NonRecoverableException: Got out-of-order key column: name: "day_of_time" type: STRING is_key: true is_nullable: false cfile_block_size: 0

原因：

[https://kudu.apache.org/docs/known_issues.html#_primary_keys](https://kudu.apache.org/docs/known_issues.html#_primary_keys)
[https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Received-ImpalaRuntimeException-Error-creating-Kudu-table/m-p/52449](https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Received-ImpalaRuntimeException-Error-creating-Kudu-table/m-p/52449)

3、kudu表的分区键必须为主键，例如执行如下语句会报错：

CREATE TABLE `t_scd_order_kudu_time` (
  `id` bigint,
  `order_no` string,
  `member_id` bigint,
  `driver_id` bigint,
  `vehicle_id` bigint,
  `origin` smallint,
  `start_city_id` int,
  `end_city_id` int,
  `estimate_board_time` string,
  `actual_board_time` string,
  `estimate_car_type` int,
  `actual_car_type` int,
  `service_type_id` smallint,
  `status` smallint,
  `balance_status` smallint,
  `balance_time` string,
  `passenger` string,
  `passenger_mobile` string,
  `comment_status` tinyint,
  `remark` string,
  `create_emp` bigint,
  `create_time` string,
  `modify_emp` bigint,
  `modify_time` string,
  `ts` string,
  `dispatch_status` smallint,
  `device_no` string,
  `ent_id` bigint,
  `invalid_type` smallint,
  `dispatch_flag` smallint,
  `share_status` smallint,
  `last_dispatch_flag` smallint,
  `home_popup` tinyint,
  `detail_popup` tinyint,
  `red_point` tinyint,
  `free_ride` tinyint,
  `ent_dept_id` bigint,
  `ent_type_id` int,
  `member_name` string,
  `member_mobile` string,
  `driver_name` string,
  `driver_mobile` string,
  `vehicle_no` string,
  `car_type_name` string,
  `ent_name` string,
  `ent_dept_name` string,
  `ent_type_name` string,
  `ent_pay_type` tinyint,
  `ent_code` string,
  `effect_response_rate` tinyint,
  `confirm_board_time` string,
  `invoice_status` smallint,
  `init_estimate_board_time` string,
  `passenger_country_code` smallint,
  `member_country_code` smallint,
  `delay_dispatch_time` int,
  `virtual_member_mobile` string,
  `virtual_passenger_mobile` string,
  `end_pl_city_code` string,
  `pl_city_code` string,
  `estimate_start_city_code` string,
  `estimate_end_city_code` string,
  `pay_mode` tinyint,
  day_of_time string,
  PRIMARY KEY (id)
)PARTITION BY HASH(day_of_time) PARTITIONS 16 STORED AS KUDU;

异常信息：

ERROR: AnalysisException: Column 'day_of_time' in 'HASH (day_of_time) PARTITIONS 16' is not a key column. Only key columns can be used in PARTITION BY.

4、impala内存溢出异常：

异常一：

Memory limit exceeded: Failed to allocate tuple buffer
HDFS_SCAN_NODE (id=0) could not allocate 2.64 MB without exceeding limit.
Error occurred on backend hadoop_datanode06_lbs:22000 by fragment 784458fdf0078462:e531b08600000002
Memory left in process limit: 1.64 MB
Query(784458fdf0078462:e531b08600000000): Reservation=0 ReservationLimit=60.40 GB OtherMemory=35.81 GB Total=35.81 GB Peak=35.81 GB
  Fragment 784458fdf0078462:e531b08600000002: Reservation=0 OtherMemory=35.81 GB Total=35.81 GB Peak=35.81 GB
    SORT_NODE (id=1): Total=35.72 GB Peak=35.72 GB
    HDFS_SCAN_NODE (id=0): Total=95.60 MB Peak=238.35 MB
    KrpcDataStreamSender (dst_id=2): Total=48.00 B Peak=48.00 B
    CodeGen: Total=80.92 KB Peak=11.54 MBProcess: Limit=75.49 GB Total=75.49 GB Peak=76.44 GB
  Buffer Pool: Free Buffers: Total=0
  Buffer Pool: Clean Pages: Total=0
  Buffer Pool: Unused Reservation: Total=-4.00 MB
  Data Stream Service Queue: Limit=3.77 GB Total=0 Peak=2.79 MB
  Data Stream Manager Early RPCs: Total=0 Peak=0
  TCMalloc Overhead: Total=98.69 MB
  RequestPool=fe-eval-exprs: Total=0 Peak=4.00 KB
  RequestPool=default-pool: Total=74.87 GB Peak=75.45 GB
    Query(784458fdf0078462:e531b08600000000): Reservation=0 ReservationLimit=60.40 GB OtherMemory=35.81 GB Total=35.81 GB Peak=35.81 GB
    Query(a4489e09d0418e18:8e8f242700000000): Reservation=4.00 MB ReservationLimit=60.40 GB OtherMemory=39.06 GB Total=39.06 GB Peak=39.07 GB
  Untracked Memory: Total=539.32 MB

异常二：

ERROR: ExecQueryFInstances rpc query_id=6e4e24d671c517d2:d9f6f13400000000 failed: Failed to get minimum memory reservation of 4.00 MB on daemon hadoop_datanode10_lbs:22000 for query 6e4e24d671c517d2:d9f6f13400000000 due to following error: Failed to increase reservation by 4.00 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=64.16 GB reservation=64.16 GB used_reservation=0 child_reservations=64.16 GB
The top 5 queries that allocated memory under this tracker are:
Query(b47f89cec2af2be:3012ffb200000000): Reservation=52.54 GB ReservationLimit=60.39 GB OtherMemory=38.42 MB Total=52.57 GB Peak=52.57 GB
Query(d84d2c69745c56e3:6698908700000000): Reservation=11.59 GB ReservationLimit=60.39 GB OtherMemory=38.33 MB Total=11.63 GB Peak=12.45 GB
Query(6e4e24d671c517d2:d9f6f13400000000): Reservation=0 ReservationLimit=60.39 GB OtherMemory=0 Total=0 Peak=0

参考链接：Impala - Memory limit exceeded

5、从hive的parquet表往kudu迁移数据时出现的主键重复警告，解决办法是改变主键列的值生成策略，警告如下：

WARNINGS: Key already present in Kudu table 'impala::kudu_test.t_scd_order_h_dt_month_range'. (1 of 3658 similar)

6、kudu新加入节点异常：

F1225 17:40:53.693409 11791 master_main.cc:77] Check failed: _s.ok() Bad status: Invalid argument: Unable to initialize catalog manager: Failed to initialize sys tables async: on-disk master list (hadoop-bi06-test1-rgtj5-tj1:7051, hadoop-bi07-test1-rgtj5-tj1:7051, hadoop20-test1-rgtj5-tj1:7051, hadoop21-test1-rgtj5-tj1:7051, hadoop22-test1-rgtj5-tj1:7051) and provided master list (hadoop20-test1-rgtj5-tj1:7051, hadoop21-test1-rgtj5-tj1:7051, hadoop22-test1-rgtj5-tj1:7051) differ. Their symmetric difference is: hadoop-bi06-test1-rgtj5-tj1:7051, hadoop-bi07-test1-rgtj5-tj1:7051

解决方案：

https://kudu.apache.org/docs/administration.html#migrate_to_multi_master

https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Adding-a-master-server-in-kudu/td-p/64612

7、impala集成kudu信息不完整报错：

Negotiation complete: Network error: Server connection negotiation failed: server connection from 10.104.132.72:12691: BlockingRecv error: Recv() got EOF from remote (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":41,"threads_started":1}

这主要是由于kudu节点改变后，impala配置文件中设置的kudu服务器地址不完全导致的，可以将/etc/default/impala中的kudu_master_hosts设置完全解决。