Hive&hadoop 安装配置

Hive&hadoop 安装配置

hadoop配置文档

  tar zxvf hadoop-0.20.2.tar.gz
  cd conf
  vim core-config.xml

增加以下内容

 
 2 
 3 
 4 
 5 
 6 
 7     
 8         fs.default.name
 9         hdfs://172.16.18.202:9000
 10     
 11 
 12     mapred.job.tracker
 13     172.16.18.202:9001
 14   
 15   
 16     dfs.replication
 17     1
 18   
 19 

怎么做授权

 172.16.60.96 需要授权guojinyong  到 172.16.60.202
 先在202 useradd guojinyong

 然后在96上 
 ssh-keygen -t rsa
 一直回车,会有一个id_rsa.pub的文件

 scp id_rsa.pub [email protected]:/tmp/
 cat /tmp/id_rsa.pub >> /home/guojinyong/.ssh/authorized_keys

这样就okey了.

hive配置文档

 配置文档
  tar -xzvf hive-x.y.z.tar.gz
  cd hive-x.y.z
  export HIVE_HOME=`pwd`
  export PATH=$HIVE_HOME/bin:$PATH

  配置hadoop 环境变量 
  export HADOOP_HOME=/work/guojinyong/hadoop-0.20.2  //假设我的hadoop解压在这里
  bin/hive
  hive> show databases;

ok搞定.

hive实战步骤

 vim /tmp/guojinyong/test/mobile.py

 import sys
 import datetime
 
 for line in sys.stdin:
   line = line.strip()
   name,num = line.split('\t')
   str1 = 'ddddd'
   str2 = 'ggggg'
   print '\t'.join([name,num,str1,str2])

 add file /tmp/guojinyong/test/mobile.py
 insert into table test select transform (str) using 'python mobile.py' as (name,num,str1,str2) from test_tmp;