lixuguang

hive 排序特性研究

1. 排序定义

所谓排序就是使一串记录，按照其中的某个或某些关键字，递增或是递减的排列。

2. hive 中排序相关内容

2.1 order by

order by 会对输入做全局排序，故只有一个reducer,若数据的规模比较大时，需要较长的计算时间。hive中order by 也是对一个结果集进行排序，不同于关系型数据库是底层架构。hive的hive-site.xml配置文件中的参数hive.mapred.mode控制着hive的执行方式，若选择strict，则order by 则需要指定limit（若有分区还有指定哪个分区）；若为nostrict，则与关系型数据库差不多。由于order by 执行时，只有一个reducer ，如果结果集过大，那执行时间相对会比较漫长。

注：若不想修改配置文件，可临时执行：set hive.mapred.mode=nonstrict 或set hive.mapred.mode=strict;也可以在当前会话中达到同样的效果。

测试：

--未开启strict模式,即nostrict模式
hive> select id,devid,devname from tb_in_base order by devid;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15431, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15431
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15431
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-08-05 16:33:21,817 Stage-1 map = 0%, reduce = 0%
2013-08-05 16:33:23,828 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.94 sec
2013-08-05 16:33:24,834 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.94 sec
2013-08-05 16:33:25,843 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.94 sec
2013-08-05 16:33:26,849 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.94 sec
2013-08-05 16:33:27,855 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.94 sec
2013-08-05 16:33:28,860 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.94 sec
2013-08-05 16:33:29,873 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.94 sec
2013-08-05 16:33:30,880 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 0.94 sec
2013-08-05 16:33:31,888 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.51 sec
2013-08-05 16:33:32,893 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.51 sec
2013-08-05 16:33:33,899 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.51 sec
MapReduce Total cumulative CPU time: 2 seconds 510 msec
Ended Job = job_201307151509_15431
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Cumulative CPU: 2.51 sec HDFS Read: 559 HDFS Write: 138 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 510 msec
OK
1 121212 test1
2 131313 test2
3 141414 test3
4 151515 test5
5 161616 test6
6 171717 test7
8 191919 test9overwrite
8 191919 test9overwrite
Time taken: 16.872 seconds

结果说明：没有开启严格模式时，order by 与关系型数据库效果类似。
--开启scrict模式，且未在order by 后面加limit
hive> select id,devid from tb_in_base order by devid;
FAILED: Error in semantic analysis: 1:41 In strict mode, if ORDER BY is specified, LIMIT must also be specified. Error encountered near token 'devid'

注：没有指定limit 报错

--开启scrict模式，且未在order by 后面加limit ，且未指定分区

hive> select * from tb_in_base;
FAILED: Error in semantic analysis: No partition predicate found for Alias "tb_in_base" Table "tb_in_base"

结果说明：严格模式下，无法直接进行查询。

--开启scrict模式，且未在order by 后面加limit ，且指定分区

hive> select * from tb_in_base where job_time=030729 order by devid limit 2;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15432, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15432
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15432
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-08-05 16:47:32,900 Stage-1 map = 0%, reduce = 0%
2013-08-05 16:47:34,920 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:35,927 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:36,934 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:37,941 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:38,946 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:39,953 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:40,959 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:41,965 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.39 sec
2013-08-05 16:47:42,971 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.05 sec
2013-08-05 16:47:43,977 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.05 sec
2013-08-05 16:47:44,983 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.05 sec
MapReduce Total cumulative CPU time: 3 seconds 50 msec
Ended Job = job_201307151509_15432
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Cumulative CPU: 3.05 sec HDFS Read: 458 HDFS Write: 44 SUCCESS
Total MapReduce CPU Time Spent: 3 seconds 50 msec
OK
1 121212 test1 030729
2 131313 test2 030729
Time taken: 17.597 seconds

结果说明：严格模式下，使用order by 不仅需要指定limit 数量，若有表分区还需要指定表分区。

2.2 sort by

sort可以控制每个reduce产生的文件都是排序，再对多个排序的好的文件做二次归并排序。sort by 特点如下：
1) . sort by 基本受hive.mapred.mode是否为strict、nonstrict的影响，但若有分区需要指定分区。
2). sort by 的数据在同一个reduce中数据是按指定字段排序。
3). sort by 可以指定执行的reduce个数，如：set mapred.reduce.tasks=5 ,对输出的数据再执行归并排序，即可以得到全部结果。

-- 在hive.mapred.mode为nonstrict时

hive> select id,devid from tb_in_base sort by devid;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15434, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15434
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15434
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-08-05 17:15:38,244 Stage-1 map = 0%, reduce = 0%
2013-08-05 17:15:40,258 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:41,269 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:42,275 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:43,281 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:44,286 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:45,292 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:46,298 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:47,305 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
2013-08-05 17:15:48,311 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.07 sec
2013-08-05 17:15:49,317 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.07 sec
2013-08-05 17:15:50,323 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.07 sec
MapReduce Total cumulative CPU time: 3 seconds 70 msec
Ended Job = job_201307151509_15434
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Cumulative CPU: 3.07 sec HDFS Read: 559 HDFS Write: 72 SUCCESS
Total MapReduce CPU Time Spent: 3 seconds 70 msec
OK
1 121212
2 131313
3 141414
4 151515
5 161616
6 171717
8 191919
8 191919
Time taken: 15.969 seconds

---开启hive.mapred.mode为strict模式
hive> set hive.mapred.mode=strict;
hive> select * from tb_in_base order by devid; //注：测试set hive.mapred.mode=strict;是否生效

FAILED: Error in semantic analysis: 1:34 In strict mode, if ORDER BY is specified, LIMIT must also be specified. Error encountered near token 'devid'

---开启hive.mapred.mode为strict模式，查询时未指定分区

hive> select * from tb_in_base sort by devid;
FAILED: Error in semantic analysis: No partition predicate found for Alias "tb_in_base" Table "tb_in_base"

---开启hive.mapred.mode为strict模式，查询时指定分区

hive> select id,devid,job_time from tb_in_base where job_time=030729 sort by devid;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15438, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15438
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15438
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-08-05 17:22:02,631 Stage-1 map = 0%, reduce = 0%
2013-08-05 17:22:05,645 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.89 sec
2013-08-05 17:22:06,651 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.89 sec
2013-08-05 17:22:07,656 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.89 sec
2013-08-05 17:22:08,662 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.89 sec
2013-08-05 17:22:09,668 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.89 sec
2013-08-05 17:22:10,673 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.89 sec
2013-08-05 17:22:11,679 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.89 sec
2013-08-05 17:22:12,685 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 0.89 sec
2013-08-05 17:22:13,691 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.61 sec
2013-08-05 17:22:14,696 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.61 sec
2013-08-05 17:22:15,704 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.61 sec
MapReduce Total cumulative CPU time: 2 seconds 610 msec
Ended Job = job_201307151509_15438
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Cumulative CPU: 2.61 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 610 msec
OK
1 121212 030729
2 131313 030729
3 141414 030729
4 151515 030729
5 161616 030729
6 171717 030729
8 191919 030729
Time taken: 18.044 seconds

结果说明：严格模式下，sort by 不指定limit 数，可以正常执行。sort by 受hive.mapred.mode=sctrict 的影响较小。

---set mapred.reduce.tasks=2，使用sort by

hive> set mapred.reduce.tasks=2;
hive> insert overwrite local directory '/tmp/hivetest/sortby' select id,devid,job_time from tb_in_base where job_time=030729 sort by devid; //注：将查询结果写入到指定目录中
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2 //注：说明有两个reducer
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15466, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15466
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15466
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 2
2013-08-05 18:03:48,298 Stage-1 map = 0%, reduce = 0%
2013-08-05 18:03:50,307 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:03:51,312 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:03:52,317 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:03:53,322 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:03:54,327 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:03:55,333 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:03:56,338 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:03:57,343 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 1.33 sec
2013-08-05 18:03:58,351 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.57 sec
2013-08-05 18:03:59,356 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.57 sec
2013-08-05 18:04:00,362 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.57 sec
MapReduce Total cumulative CPU time: 4 seconds 570 msec
Ended Job = job_201307151509_15466
Copying data to local directory /tmp/hivetest/sortby
Copying data to local directory /tmp/hivetest/sortby
7 Rows loaded to /tmp/hivetest/sortby
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 2 Cumulative CPU: 4.57 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 570 msec
OK
Time taken: 16.712 seconds

查看/tmp/hivetest/sort下查询结果

结果说明：sort by 可以通过mapred.reduce.task指定reduce个数，可以将查询后的数据分别写入两个不同的reduce文件中。

---set mapred.reduce.tasks=2，使用order by

hive> set mapred.reduce.tasks=2;
hive> insert overwrite local directory '/tmp/hivetest/orderby' select id,devid,job_time from tb_in_base where job_time=030729 order by devid;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1 //注：说明只有一个reducer
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15469, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15469
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15469
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-08-05 18:14:22,528 Stage-1 map = 0%, reduce = 0%
2013-08-05 18:14:24,538 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2013-08-05 18:14:25,548 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2013-08-05 18:14:26,562 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2013-08-05 18:14:27,568 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2013-08-05 18:14:28,574 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2013-08-05 18:14:29,581 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2013-08-05 18:14:30,587 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2013-08-05 18:14:31,592 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 1.05 sec
2013-08-05 18:14:32,598 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.56 sec
2013-08-05 18:14:33,604 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.56 sec
2013-08-05 18:14:34,611 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.56 sec
MapReduce Total cumulative CPU time: 2 seconds 560 msec
Ended Job = job_201307151509_15469
Copying data to local directory /tmp/hivetest/orderby
Copying data to local directory /tmp/hivetest/orderby
7 Rows loaded to /tmp/hivetest/orderby
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Cumulative CPU: 2.56 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 560 msec
OK
Time taken: 16.804 seconds

查看/tmp/hivetest/orderby下查询结果

结果说明：设置mapred.reduce.task=2，使用order by做排序，所有的数据都是写入同一个文件中。由此可说明不论设计多少个reduce任务数，order by 只使用一个reduce。

注：可以用limit 子句大大减少数据量，使用limit n ，后，传输到reduce商的数据记录就减少到n* map，否则可能由于数据过大可能出不了结果。

疑问：使用limit n之后，数据是减少了，但是对于统计数据的话，少了数据的统计还有意义吗？

2.3 distribute by

distribute by 是控制在map端如何拆分给reduce端。根据distribute by 后面的列及reduce个数进行数据分发，默认采用hash算法。distribute可以使用length方法会根据string类型的长度划分到不同的reduce中，最终输出到不同的文件中。 length 是内建函数，也可以指定其他的函数或这使用自定义函数。

注：对distribute by 进行测试，一定要分配多个reduce进行处理，否则无法看到distribute by 效果。

--使用id进行排序且set mapred.reduce.tasks=2;

hive> set mapred.reduce.tasks=2;
hive> insert overwrite local directory '/tmp/hivetest/distributeby' select id,devid,job_time from tb_in_base where job_time=030729 and id<8 distribute by id ;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15499, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15499
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15499
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 2
2013-08-05 18:37:14,681 Stage-1 map = 0%, reduce = 0%
2013-08-05 18:37:16,691 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:37:17,697 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:37:18,703 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:37:19,710 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:37:20,717 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:37:21,727 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:37:22,733 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
2013-08-05 18:37:23,739 Stage-1 map = 100%, reduce = 50%, Cumulative CPU 3.1 sec
2013-08-05 18:37:24,745 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.89 sec
2013-08-05 18:37:25,751 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.89 sec
2013-08-05 18:37:26,757 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.89 sec
MapReduce Total cumulative CPU time: 4 seconds 890 msec
Ended Job = job_201307151509_15499
Copying data to local directory /tmp/hivetest/distributeby
Copying data to local directory /tmp/hivetest/distributeby
7 Rows loaded to /tmp/hivetest/distributeby
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 2 Cumulative CPU: 4.89 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 890 msec
OK
Time taken: 16.785 seconds

查看写入的查询数据:

结果说明：distribute by采用hash算法，将查询的结果写入不同的reduce文件中。数据分配到哪个reduce文件中，是在map端控制的。

--使用job_time进行排序且set mapred.reduce.tasks=2;

hive> set mapred.reduce.tasks=2;
hive> insert overwrite local directory '/tmp/hivetest/distributeby' select id,devid,job_time from tb_in_base where job_time=030729 distribute by job_time;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15500, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15500
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15500
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 2
2013-08-05 18:42:07,764 Stage-1 map = 0%, reduce = 0%
2013-08-05 18:42:10,778 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.61 sec
2013-08-05 18:42:11,784 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.61 sec
2013-08-05 18:42:12,789 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.61 sec
2013-08-05 18:42:13,795 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.61 sec
2013-08-05 18:42:14,801 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.61 sec
2013-08-05 18:42:15,810 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.61 sec
2013-08-05 18:42:16,816 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.61 sec
2013-08-05 18:42:17,821 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 0.61 sec
2013-08-05 18:42:18,827 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.89 sec
2013-08-05 18:42:19,833 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.89 sec
2013-08-05 18:42:20,839 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.89 sec
MapReduce Total cumulative CPU time: 3 seconds 890 msec
Ended Job = job_201307151509_15500
Copying data to local directory /tmp/hivetest/distributeby
Copying data to local directory /tmp/hivetest/distributeby
7 Rows loaded to /tmp/hivetest/distributeby
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 2 Cumulative CPU: 3.89 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 3 seconds 890 msec
OK
Time taken: 16.678 seconds

查看查询后的结果：

结果说明：hash值一样的数据被分配到同一个reduce中。

额外验证:
---sort by 是否也像distribute by 一样使用hash算法

set mapred.reduce.tasks=2;
hive> insert overwrite local directory '/tmp/hivetest/sortby' select id,devid,job_time from tb_in_base where job_time=030729 sort by id;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15501, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15501
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15501
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 2
2013-08-05 18:57:33,616 Stage-1 map = 0%, reduce = 0%
2013-08-05 18:57:35,625 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:36,631 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:37,636 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:38,642 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:39,648 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:40,653 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:41,659 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:42,669 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.16 sec
2013-08-05 18:57:43,675 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 3.02 sec
2013-08-05 18:57:44,681 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.68 sec
2013-08-05 18:57:45,687 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.68 sec
2013-08-05 18:57:46,693 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.68 sec
MapReduce Total cumulative CPU time: 4 seconds 680 msec
Ended Job = job_201307151509_15501
Copying data to local directory /tmp/hivetest/sortby
Copying data to local directory /tmp/hivetest/sortby
7 Rows loaded to /tmp/hivetest/sortby
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 2 Cumulative CPU: 4.68 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 680 msec
OK

查询结果：

结果说明：sort by 不支持通过hash算法将数据分配到不同的reduce文件。

2.4 cluster by

cluster by 除了distribute by 的功能外，还会对该字段进行排序，所以cluster by = distribute by +sort by

---cluster by 是否可以指定asc 或desc

hive> select id,devid,job_time from tb_in_base where job_time=030729 cluster by id desc;
FAILED: Parse Error: line 1:77 mismatched input 'desc' expecting EOF near 'id'

hive> select id,devid,job_time from tb_in_base where job_time=030729 cluster by id asc;
FAILED: Parse Error: line 1:77 mismatched input 'asc' expecting EOF near 'id'

注：cluster by 默认倒序排序

hive> set mapred.reduce.tasks=2;
hive> insert overwrite local directory '/tmp/hivetest/clusterby' select id,devid,job_time from tb_in_base where job_time=030729 cluster by id;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15532, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15532
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15532
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 2
2013-08-05 19:41:15,138 Stage-1 map = 0%, reduce = 0%
2013-08-05 19:41:17,147 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:18,153 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:19,158 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:20,163 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:21,169 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:22,174 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:23,180 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:24,186 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
2013-08-05 19:41:25,193 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 2.5 sec
2013-08-05 19:41:26,199 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.34 sec
2013-08-05 19:41:27,205 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.34 sec
2013-08-05 19:41:28,210 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.34 sec
MapReduce Total cumulative CPU time: 4 seconds 340 msec
Ended Job = job_201307151509_15532
Copying data to local directory /tmp/hivetest/clusterby
Copying data to local directory /tmp/hivetest/clusterby
7 Rows loaded to /tmp/hivetest/clusterby
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 2 Cumulative CPU: 4.34 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 340 msec
OK
Time taken: 17.637 seconds

写入文件后的查询结果：

结果分析：采用hash算法，根据hash值将查询的结果写入不同的reduce文件中。

hive> set mapred.reduce.tasks=2;
hive> insert overwrite local directory '/tmp/hivetest/clusterby' select id,devid,job_time from tb_in_base where job_time=030729 cluster by job_time;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307151509_15533, Tracking URL = http://mwtec-50:50030/jobdetails.jsp?jobid=job_201307151509_15533
Kill Command = /home/hadoop/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=mwtec-50:9002 -kill job_201307151509_15533
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 2
2013-08-05 19:43:38,722 Stage-1 map = 0%, reduce = 0%
2013-08-05 19:43:40,732 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.5 sec
2013-08-05 19:43:41,738 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.5 sec
2013-08-05 19:43:42,743 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.5 sec
2013-08-05 19:43:43,748 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.5 sec
2013-08-05 19:43:44,754 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.5 sec
2013-08-05 19:43:45,759 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.5 sec
2013-08-05 19:43:46,765 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.5 sec
2013-08-05 19:43:47,770 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 1.5 sec
2013-08-05 19:43:48,776 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 3.02 sec
2013-08-05 19:43:49,781 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.55 sec
2013-08-05 19:43:50,787 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.55 sec
MapReduce Total cumulative CPU time: 4 seconds 550 msec
Ended Job = job_201307151509_15533
Copying data to local directory /tmp/hivetest/clusterby
Copying data to local directory /tmp/hivetest/clusterby
7 Rows loaded to /tmp/hivetest/clusterby
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 2 Cumulative CPU: 4.55 sec HDFS Read: 458 HDFS Write: 112 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 550 msec
OK
Time taken: 16.613 seconds

写入文件后的查询结果：

结果说明：cluster by 对其指定的列，进行hash，若hash值相同则被分配到相同的reduce文件中。

3 . 总结分析

1). order by 只有一个reduce负责对所有的数据进行排序，若大数据量，则需要较长的时间。建议在小的数据集中使用order by 进行排序。

2). order by 可以通过设置hive.mapred.mode参数控制执行方式，若选择strict，则order by 则需要指定limit（若有分区还有指定哪个分区）；若为nostrict，则与关系型数据库差不多。

3). sort by 基本上不受hive.mapred.mode影响，可以通过mapred.reduce.task 指定reduce个数，查询后的数据被分发到相关的reduce中。

4). sort by 的数据在进入reduce前就完成排序，如果要使用sort by 是行排序，并且设置map.reduce.tasks>1,则sort by 才能保证每个reducer输出有序，不能保证全局数据有序。

5). distribute by 采集hash算法，在map端将查询的结果中hash值相同的结果分发到对应的reduce文件中。

6). distribute by 可以使用length方法会根据string类型的长度划分到不同的reduce中，最终输出到不同的文件中。 length 是内建函数，也可以指定其他的函数或这使用自定义函数。

7). cluster by 除了distribute by 的功能外，还会对该字段进行排序，所以cluster by = distribute by +sort by 。

你可能感兴趣的:(hive)

nosql数据库技术与应用知识点皆过客，揽星河 NoSQL nosql 数据库大数据数据分析数据结构非关系型数据库
Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
浅谈MapReduce Android路上的人 Hadoop 分布式计算 mapreduce 分布式框架 hadoop
从今天开始，本人将会开始对另一项技术的学习，就是当下炙手可热的Hadoop分布式就算技术。目前国内外的诸多公司因为业务发展的需要，都纷纷用了此平台。国内的比如BAT啦，国外的在这方面走的更加的前面，就不一一列举了。但是Hadoop作为Apache的一个开源项目，在下面有非常多的子项目，比如HDFS，HBase,Hive，Pig,等等，要先彻底学习整个Hadoop，仅仅凭借一个的力量，是远远不够的。
Presto【基础 01】简介+架构+数据源+数据模型 2401_84254343 程序员架构
一个Catalog包含Schema和Connector。例如，配置JMX的Catalog，通过JXMConnector访问JXM信息。当执行一条SQL语句时，可以同时运行在多个Catalog。Presto处理table时，是通过表的完全限定（fully-qualified）名来找到Catalog。例如，一个表的权限定名是hive.test_data.test，则test是表名，test_data是
大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏租房推荐系统 58同城租房爬虫房源推荐系统房价预测系统计算机毕业设计机器学习深度学习人工智能 2401_84572577 程序员大数据 hadoop 人工智能
做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
大数据之flink与hive 星辰_mya 大数据 flink hive
其实吧我不太想写flink，因为线上经验确实不多，这也是我需要补的地方，没有条件创造条件，先来一篇吧flink：高性能低延迟流批一体的分布式计算框架基于事件时间对实时数据精准处理快速响应支持批处理，高效离线分析和数据挖掘数据仓库的引擎丰富数据源/接收器，集成多种数据存储格式和源，比较常见就是咱们今天的主题hive了checkpoint恢复机制，故障恢复快速恢复计算任务分布式弹性扩展，据业务灵活增加
hive血缘关系之输入表与目标表的解析 zxfBdd hive 大数据治理大数据
接了一个新需求：需要做数据仓库的血缘关系。正所谓兵来将挡水来土掩，那咱就动手吧。血缘关系是数据治理的一块，其实有专门的第三方数据治理框架，但考虑到目前的线上环境已经趋于稳定，引入新的框架无疑是劳民伤财，伤筋动骨，所以就想以最小的代价把这个事情给做了。目前我们考虑做的血缘关系呢只是做输入表和输出表，最后会形成一张表与表之间的链路图。这个东西的好处就是有助于仓库人员梳理业务，后面可能还会做字段之间的血
初级练习[3]:Hive SQL子查询应用大数据深度洞察 Hive hive sql hadoop 数据仓库大数据数据库
目录环境准备看如下链接子查询查询所有课程成绩均小于60分的学生的学号、姓名查询没有学全所有课的学生的学号、姓名解释：没有学全所有课，也就是该学生选修的课程数<总的课程数。查询出只选修了三门课程的全部学生的学号和姓名环境准备看如下链接环境准备https://blog.csdn.net/qq_45115959/article/details/142057624?spm=1001.2014.3001.5
Linux下载压缩包：tar.gz、zip、tar.bz2格式全攻略 promise524 Linux linux 运维服务器后端 bash shell
在Linux中，下载各种格式的压缩包（如.tar.gz、.zip、.tar.bz2等）通常使用命令行工具如wget和curl。1.使用wget下载压缩包wget是Linux中最常用的文件下载工具，支持HTTP、HTTPS、FTP等协议，可以直接从命令行下载文件。基本命令：wget[URL]下载.tar.gz文件wgethttps://test.com/archive.tar.gz此命令将从指定的U
Anaconda版本和Python版本对应关系纬领网络 python anaconda3
官网下载地址：https://repo.anaconda.com/archive/下载地址：https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/anaconda3版本基础python版本Anaconda3-2024.06-1Python3.12.4Anaconda3-2024.02-1Python3.11.7Anaconda3-2023.09
R语言包AMORE安装报错问题以及RStudio与Rtools环境配置卡卡_R-Python R语言数据分析与可视化 r语言开发语言
在使用R语言进行AMORE安装时会遇到报错，这时候需要采用解决办法：'''AMORE包安装，需要离线官网下载安装包：Indexof/src/contrib/Archive/AMORE(r-project.org)https://cran.r-project.org/src/contrib/Archive/AMORE/一、出现的问题最近开始学习R语言，安装了最新版的R4.4.1和RStudio，但安
中级练习[3]：Hive SQL用户行为与商品销售数据分析大数据深度洞察 Hive hive 数据仓库大数据 sql
目录1.用户累计消费金额及VIP等级查询1.1题目需求1.2代码实现2.首次下单后第二天连续下单的用户比率查询2.1题目需求2.2代码实现3.每个商品销售首年的年份、销售数量和销售金额统计3.1题目需求3.2代码实现1.用户累计消费金额及VIP等级查询1.1题目需求从订单信息表(order_info)中统计每个用户截止其每个下单日期的累积消费金额，以及每个用户在其每个下单日期的VIP等级。VIP等
Python基础知识进阶之正则表达式_头歌python正则表达式进阶前端陈萨龙程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
编程常用命令总结 Yellow0523 Linux BigData 大数据
编程命令大全1.软件环境变量的配置JavaScalaSparkHadoopHive2.大数据软件常用命令Spark基本命令Spark-SQL命令Hive命令HDFS命令YARN命令Zookeeper命令kafka命令Hibench命令MySQL命令3.Linux常用命令Git命令conda命令pip命令查看Linux系统的详细信息查看Linux系统架构(X86还是ARM，两种方法都可)端口号命令L
博客园怎么了？ YYH1992
新年好，给大家拜个早年！今年来到安徽过年，无聊中，不知不觉中又来到博客园了（忠实粉丝哦），却发现一件奇怪的事情，请看截图难道博客园被挂马了？抑或其它问题？如果真有问题，还请dudu抓紧时间修正，免得影响我们园子的声誉！我要下线了，出去买回家的车票了，只能年后回家了。。。转载于:https://www.cnblogs.com/HollisYao/archive/2008/02/06/1065351.
linux下文件的复制、移动与删除搬砖中年人
一、文件复制命令cp命令格式：cp[-adfilprsu]源文件(source)目标文件(destination)cp[option]source1source2source3...directory参数说明：-a:是指archive的意思，也说是指复制所有的目录-d:若源文件为连接文件(linkfile)，则复制连接文件属性而非文件本身-f:强制(force)，若有重复或其它疑问时，不会询问用户
2024年最全使用Python求解方程_python解方程(1)，字节面试官迟到 2401_84569545 程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
兼容 Trino Connector，扩展 Apache Doris 数据源接入能力｜Lakehouse 使用手册 vvvae1234 apache
ApacheDoris内置支持包括Hive、Iceberg、Hudi、Paimon、LakeSoul、JDBC在内的多种Catalog，并为其提供原生高性能且稳定的访问能力，以满足与数据湖的集成需求。而随着ApacheDoris用户的增加，新的数据源连接需求也随之增加。因此，从3.0版本开始，ApacheDoris引入了TrinoConnector兼容框架。Trino/Presto作为业界较早应用
SAP HANA makaitai BW sap 数据库工具报表 layer 服务器
原文地址：http://LiuAlex.com/archives/1776也是刚刚开始学习HANA的一些知识，一边看书一遍做笔记，说到底无非是用自己的语言来理解标准帮组文档所讲解的意思，肯定有理解失误的地方，毕竟没有参加过标准培训，即使有培训，从老师那边来的知识也不可能是完整的传授过来，中间多少的知识遗漏是正常的，所以多看看HELP的文档，应该可以原汁原味的理解作者的意思。这张图片是从SAPHAN
Hive SQL查询汇总分析大数据深度洞察 Hive hive sql hadoop 数据仓库数据库大数据
目录SQL查询汇总分析成绩查询查询编号为“02”的课程的总成绩查询参加考试的学生个数分组查询查询各科成绩最高和最低的分查询每门课程有多少学生参加了考试（有考试成绩）查询男生、女生人数分组结果的条件查询平均成绩大于60分的学生的学号和平均成绩查询至少选修四门课程的学生学号查询同姓（假设每个学生姓名的第一个字为姓）的学生名单并统计同姓人数大于2的姓查询每门课程的平均成绩，结果按平均成绩升序排序，平均成
RMAN-08137 rman delete archivelog force jnrjian 数据库 oracle
deleteforcearchiveloguntiltime'trunc(sysdate-4)'backedup1timestodevicetypedisk;SymptomsDatabaseAClonedtoDatabaseBonCloneserver.GoldenGateisConfiguredonSourcedatbaseA.DatabaseBwhichisclonedfromSourcedo
hive表格统计信息不准确 weixin_41956627 hive hive hadoop 数据仓库
问题描述有个hive分区表，orc存储格式，有个分区，查询selectcount(1)fromtablewheredt='yyyyMMdd'结果是0，但查询select*fromtablewheredt='yyyyMMdd'又能查到数据，去hdfs对应目录下查看，也能看到有数据文件解决执行如下sqlANALYZETABLEdb.table1PARTITION(dt='20240908')COMPU
Conda创建环境失败：000和404错误柚柚柚柚柚 conda
一、首先下载Anaconda1.打开网址Indexof/anaconda/archive/|清华大学开源软件镜像站|TsinghuaOpenSourceMirror，滑到最底部，下载Anaconda3-5.3.1-Linux-x86_64.sh。2.使用winscp拖动本地的Anaconda3-5.3.1-Linux-x86_64.sh到服务器的个人工作目录下。二、安装Anaconda软件，创建虚
C#中两个问号的含义 weixin_30363981 测试
stringstrParam=Request.Params["param"]??"";取??左边的值,如果??左边的值为null则取右边的值转载于:https://www.cnblogs.com/shadowtale/archive/2012/10/19/2731152.html
如何下载各个版本的tomcat-比如tomcat9 耳边轻语999 tomcat java
1，找到tomcat官网https://tomcat.apache.org/ApacheTomcat®-Welcome!找到tomcat9，或者archives1.1，找到对应版本1.2，找到小版本1.3，找到bin2，Indexof/dist/tomcat/tomcat-9/v9.0.39/bin2.1，下载对应的解压版本或者安装版本
Percona-toolkit工具详解小一_d28d
1.pt工具安装[root@master~]#yuminstall-ypercona-toolkit-3.1.0-2.el7.x86_64.rpm2.常用工具使用介绍2.1pt-archiver归档表#重要参数--limit100每次取100行数据用pt-archive处理--txn-size100设置100行为一个事务提交一次，--where'id>/root/db/checksum.logpt
Ubuntu更换apt-get的下载源愤愤的有痣青年
将以下内容替换/etc/apt/sources.list中的内容deb-srchttp://archive.ubuntu.com/ubuntuxenialmainrestricted#Addedbysoftware-propertiesdebhttp://mirrors.aliyun.com/ubuntu/xenialmainrestricteddeb-srchttp://mirrors.aliy
apt 下载指定架构的包及离线安装的方法错误重复学习记录 linux
#设置系统架构sudodpkg--add-architectureamd64#安装apt-rdependssudoaptinstallapt-rdepends#创建单独的目录mkdir-p/home/apt/postgresql-client-common#仅下载安装包sudoapt-getinstall--download-onlysudomv/var/cache/apt/archives/*/
游戏运营环节的一些关键转化率 turtle081025 数据分析游戏网络游戏运营
转载于http://www.gamedatas.com/archives/134转化率这个指标在各行各业的数据分析中运用的非常之广泛，例如：电商中就会存在，点击到订单生成的一系列转化率，传统的销售行业也会在做广告的时候考虑该广告能够转化多少订单，而在游戏行业，转化率同样是一个不容忽视的指标。一般来说，游戏运营的过程中主要会关注到这些转化率：1.下载-安装（激活）转化率；2.安装（激活）-注册转化率
Python API操作RocketMQ 京城小筑 #Python编程 python
背景：开发背景:公司相关报表需求需要将订单业务数据同步至RocketMQ中，由于需要保证开发的一致性(多个部门协同开发)，所以采用读取Hive离线数据的方式通过PythonAPI写入RocketMQ中，便于其他开发同事调用~开发环境:本地调试系统MacPython3.7.5rocketmq0.4.4(Python模块)rocketmq-client-python2.0.0(Python模块)服务器
hive搭建 -----内嵌模式和本地模式 lzhlizihang hive hadoop
文章目录一、内嵌模式（使用较少）1、上传、解压、重命名2、配置环境变量3、配置conf下的hive-env.sh4、修改conf下的hive-site.xml5、启动hadoop集群6、给hdfs创建文件夹7、修改hive-site.xml中的非法字符8、初始化元数据9、测试是否成功10、内嵌模式的缺点二、本地模式（最常用）1、检查mysql是否正常2、上传、解压、重命名3、配置环境变量4、修改c
解线性方程组 qiuwanchi
package gaodai.matrix; import java.util.ArrayList; import java.util.List; import java.util.Scanner; public class Test { public static void main(String[] args) { Scanner scanner = new Sc
在mysql内部存储代码 annan211 性能 mysql 存储过程触发器
在mysql内部存储代码在mysql内部存储代码，既有优点也有缺点，而且有人倡导有人反对。先看优点： 1 她在服务器内部执行，离数据最近，另外在服务器上执行还可以节省带宽和网络延迟。 2 这是一种代码重用。可以方便的统一业务规则，保证某些行为的一致性，所以也可以提供一定的安全性。 3 可以简化代码的维护和版本更新。 4 可以帮助提升安全，比如提供更细
Android使用Asynchronous Http Client完成登录保存cookie的问题 hotsunshine android
Asynchronous Http Client是android中非常好的异步请求工具除了异步之外还有很多封装比如json的处理，cookie的处理引用 Persistent Cookie Storage with PersistentCookieStore This library also includes a PersistentCookieStore whi
java面试题 Array_06 java 面试
java面试题第一，谈谈final, finally, finalize的区别。 final-修饰符（关键字）如果一个类被声明为final，意味着它不能再派生出新的子类，不能作为父类被继承。因此一个类不能既被声明为 abstract的，又被声明为final的。将变量或方法声明为final，可以保证它们在使用中不被改变。被声明为final的变量必须在声明时给定初值，而在以后的引用中只能
网站加速 oloz 网站加速
前序:本人菜鸟，此文研究总结来源于互联网上的资料，大牛请勿喷！本人虚心学习，多指教. 1、减小网页体积的大小，尽量采用div+css模式，尽量避免复杂的页面结构，能简约就简约。 2、采用Gzip对网页进行压缩； GZIP最早由Jean-loup Gailly和Mark Adler创建，用于UNⅨ系统的文件压缩。我们在Linux中经常会用到后缀为.gz
正确书写单例模式随意而生 java 设计模式单例
　　单例模式算是设计模式中最容易理解，也是最容易手写代码的模式了吧。但是其中的坑却不少，所以也常作为面试题来考。本文主要对几种单例写法的整理，并分析其优缺点。很多都是一些老生常谈的问题，但如果你不知道如何创建一个线程安全的单例，不知道什么是双检锁，那这篇文章可能会帮助到你。　　懒汉式，线程不安全　　当被问到要实现一个单例模式时，很多人的第一反应是写出如下的代码，包括教科书上也是这样
单例模式香水浓 java
懒汉调用getInstance方法时实例化 public class Singleton { private static Singleton instance; private Singleton() {} public static synchronized Singleton getInstance() { if(null == ins
安装Apache问题：系统找不到指定的文件 No installed service named "Apache2" AdyZhang apache http server
安装Apache问题：系统找不到指定的文件 No installed service named "Apache2" 每次到这一步都很小心防它的端口冲突问题，结果，特意留出来的80端口就是不能用，烦。解决方法确保几处： 1、停止IIS启动 2、把端口80改成其它（譬如90，800，，，什么数字都好） 3、防火墙(关掉试试) 在运行处输入 cmd 回车，转到apa
如何在android 文件选择器中选择多个图片或者视频？ aijuans android
我的android app有这样的需求，在进行照片和视频上传的时候，需要一次性的从照片/视频库选择多条进行上传但是android原生态的sdk中，只能一个一个的进行选择和上传。我想知道是否有其他的android上传库可以解决这个问题，提供一个多选的功能，可以使checkbox之类的，一次选择多个处理方法官方的图片选择器(但是不支持所有版本的androi，只支持API Level
mysql中查询生日提醒的日期相关的sql baalwolf mysql
SELECT sysid,user_name,birthday,listid,userhead_50,CONCAT(YEAR(CURDATE()),DATE_FORMAT(birthday,'-%m-%d')),CURDATE(), dayofyear( CONCAT(YEAR(CURDATE()),DATE_FORMAT(birthday,'-%m-%d')))-dayofyear(
MongoDB索引文件破坏后导致查询错误的问题 BigBird2012 mongodb
问题描述： MongoDB在非正常情况下关闭时，可能会导致索引文件破坏，造成数据在更新时没有反映到索引上。解决方案：使用脚本，重建MongoDB所有表的索引。 var names = db.getCollectionNames(); for( var i in names ){ var name = names[i]; print(name);
Javascript Promise bijian1013 JavaScript Promise
Parse JavaScript SDK现在提供了支持大多数异步方法的兼容jquery的Promises模式，那么这意味着什么呢，读完下文你就了解了。一.认识Promises “Promises”代表着在javascript程序里下一个伟大的范式，但是理解他们为什么如此伟大不是件简
[Zookeeper学习笔记九]Zookeeper源代码分析之Zookeeper构造过程 bit1129 zookeeper
Zookeeper重载了几个构造函数，其中构造者可以提供参数最多，可定制性最多的构造函数是 public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher, long sessionId, byte[] sessionPasswd, boolea
【Java命令三】jstack bit1129 jstack
jstack是用于获得当前运行的Java程序所有的线程的运行情况(thread dump），不同于jmap用于获得memory dump [hadoop@hadoop sbin]$ jstack Usage: jstack [-l] <pid> (to connect to running process) jstack -F
jboss 5.1启停脚本　动静分离部署 ronin47
以前启动jboss，往各种xml配置文件，现只要运行一句脚本即可。start nohup sh /**/run.sh -c servicename -b ip -g clustername -u broatcast jboss.messaging.ServerPeerID=int -Djboss.service.binding.set=p
UI之如何打磨设计能力? brotherlamp UI ui教程 ui自学 ui资料 ui视频
在越来越拥挤的初创企业世界里，视觉设计的重要性往往可以与杀手级用户体验比肩。在许多情况下，尤其对于 Web 初创企业而言，这两者都是不可或缺的。前不久我们在《右脑革命：别学编程了，学艺术吧》中也曾发出过重视设计的呼吁。如何才能提高初创企业的设计能力呢?以下是 9 位创始人的体会。 1.找到自己的方式如果你是设计师，要想提高技能可以去设计博客和展示好设计的网站如D-lists或
三色旗算法 bylijinnan java 算法
import java.util.Arrays; /** 问题：假设有一条绳子，上面有红、白、蓝三种颜色的旗子，起初绳子上的旗子颜色并没有顺序，您希望将之分类，并排列为蓝、白、红的顺序，要如何移动次数才会最少，注意您只能在绳子上进行这个动作，而且一次只能调换两个旗子。网上的解法大多类似：在一条绳子上移动，在程式中也就意味只能使用一个阵列，而不使用其它的阵列来
警告:No configuration found for the specified action: \'s chiangfai configuration
1.index.jsp页面form标签未指定namespace属性。  <%@taglib prefix="s" uri="/struts-tags"%> ... <s:form action="submit" method="post"&g
redis -- hash_max_zipmap_entries设置过大有问题 chenchao051 redis hash
使用redis时为了使用hash追求更高的内存使用率，我们一般都用hash结构，并且有时候会把hash_max_zipmap_entries这个值设置的很大，很多资料也推荐设置到1000，默认设置为了512，但是这里有个坑 #define ZIPMAP_BIGLEN 254 #define ZIPMAP_END 255 /* Return th
select into outfile access deny问题 daizj mysql txt 导出数据到文件
本文转自：http://hatemysql.com/2010/06/29/select-into-outfile-access-deny%E9%97%AE%E9%A2%98/ 为应用建立了rnd的帐号，专门为他们查询线上数据库用的，当然，只有他们上了生产网络以后才能连上数据库，安全方面我们还是很注意的，呵呵。授权的语句如下： grant select on armory.* to rn
phpexcel导出excel表简单入门示例 dcj3sjt126com PHP Excel phpexcel
<?php error_reporting(E_ALL); ini_set('display_errors', TRUE); ini_set('display_startup_errors', TRUE); if (PHP_SAPI == 'cli') die('This example should only be run from a Web Brows
美国电影超短200句 dcj3sjt126com 电影
1. I see．我明白了。2. I quit! 我不干了!3. Let go! 放手!4. Me too．我也是。5. My god! 天哪!6. No way! 不行!7. Come on．来吧(赶快)8. Hold on．等一等。9. I agree。我同意。10. Not bad．还不错。11. Not yet．还没。12. See you．再见。13. Shut up!
Java访问远程服务 dyy_gusi httpclient webservice get post
随着webService的崛起，我们开始中会越来越多的使用到访问远程webService服务。当然对于不同的webService框架一般都有自己的client包供使用，但是如果使用webService框架自己的client包，那么必然需要在自己的代码中引入它的包，如果同时调运了多个不同框架的webService，那么就需要同时引入多个不同的clien
Maven的settings.xml配置 geeksun settings.xml
settings.xml是Maven的配置文件，下面解释一下其中的配置含义： settings.xml存在于两个地方： 1.安装的地方：$M2_HOME/conf/settings.xml 2.用户的目录：${user.home}/.m2/settings.xml 前者又被叫做全局配置，后者被称为用户配置。如果两者都存在，它们的内容将被合并，并且用户范围的settings.xml优先。
ubuntu的init与系统服务设置 hongtoushizi ubuntu
转载自： http://iysm.net/?p=178 init Init是位于/sbin/init的一个程序，它是在linux下，在系统启动过程中，初始化所有的设备驱动程序和数据结构等之后，由内核启动的一个用户级程序，并由此init程序进而完成系统的启动过程。 ubuntu与传统的linux略有不同，使用upstart完成系统的启动，但表面上仍维持init程序的形式。运行
跟我学Nginx+Lua开发目录贴 jinnianshilongnian nginx lua
使用Nginx+Lua开发近一年的时间，学习和实践了一些Nginx+Lua开发的架构，为了让更多人使用Nginx+Lua架构开发，利用春节期间总结了一份基本的学习教程，希望对大家有用。也欢迎谈探讨学习一些经验。目录第一章安装Nginx+Lua开发环境第二章 Nginx+Lua开发入门第三章 Redis/SSDB+Twemproxy安装与使用第四章 L
php位运算符注意事项 home198979 位运算 PHP &
$a = $b = $c = 0; $a & $b = 1; $b | $c = 1 问a,b,c最终为多少? 当看到这题时，我犯了一个低级错误，误以为位运算符会改变变量的值。所以得出结果是1 1 0 但是位运算符是不会改变变量的值的，例如： $a=1;$b=2; $a&$b; 这样a,b的值不会有任何改变
Linux shell数组建立和使用技巧 pda158 linux
1.数组定义　　[chengmo@centos5 ~]$ a=(1 2 3 4 5) 　　[chengmo@centos5 ~]$ echo $a 　　1 　　一对括号表示是数组，数组元素用“空格”符号分割开。　　 2.数组读取与赋值　　得到长度：　　[chengmo@centos5 ~]$ echo ${#a[@]} 　　5 　　用${#数组名[@或
hotspot源码(JDK7) ol_beta java HotSpot jvm
源码结构图，方便理解： ├─agent Serviceab
Oracle基本事务和ForAll执行批量DML练习 vipbooks oracle sql
基本事务的使用：从账户一的余额中转100到账户二的余额中去，如果账户二不存在或账户一中的余额不足100则整笔交易回滚 select * from account; -- 创建一张账户表 create table account( -- 账户ID id number(3) not null, -- 账户名称 nam