hive运维记录20111117

1）运行hive cli客户端时报错：

Mkdirs failed to create /tmp/hadoop-test/hadoop-unjar15513

    错误原因：hive客户端所在服务器/tmp目录已满

    解决办法：a）清空/tmp目录；b）修改hive的临时目录

2）hive提交的作业始终是map 0% red 0%的状态：

    作业图示如下：

  

   发现该job分配的map数为零，最初还以为是jobtracker的问题（因为刚上了fair scheduler，且恰巧昨天该job运行时，jobtracker也抛出了异常），但是今天复现时，发现其他的mapred job都能够正常运行，所以在仔细查看了日志后发现：

2011-11-17 10:17:36,071 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201111170937_0037 = 0. Number of splits = 0
2011-11-17 10:17:36,071 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201111170937_0037 initialized successfully with 0 map tasks and 1 reduce tasks.

  该作业的输入数据分片为0，所以jobtracker分配了零个map，一个red槽位，从而导致了该作业始终是map 0% red 0%的状态，不知道这算不算是mapred的一个bug

   3）在安装新的hive服务时，必须要在hosts文件中添加hadoop集群服务器的ip hostname信息，且还需要添加本机的ip hostname信息，否则会抛出以下错误：

java.net.UnknownHostException: test: test
        at java.net.InetAddress.getLocalHost(InetAddress.java:1354)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:815)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765)
        at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:657)
        at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Job Submission failed with exception 'java.net.UnknownHostException(test: test)'

     4）hive内存分配错误信息：

Exception in thread "pool-1-thread-4345" java.lang.OutOfMemoryError: Java heap space

    5）hive查看hadoop配置：

set -v 可得到所有环境变量。如果没有-v参数，只显示与hadoop不同的配置。