Sqoop 1.99.3 with hadoop-2.3.0 使用 3

使用压缩来看看

 

sqoop:000> update job --jid 1

Compression format:

  0 : NONE

  1 : DEFAULT

  2 : DEFLATE

  3 : GZIP

  4 : BZIP2

  5 : LZO

  6 : LZ4

  7 : SNAPPY

Choose: 3

Output directory: /home/dimDateGZip

Job was successfully updated with status FINE

 

使用Gzip

同样的job 跑出来的不一样

 

[root@localhost ~]# hadoop fs -ls /home/dimDateGZip
14/03/20 09:39:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 11 items
-rw-r--r--   1 root supergroup          0 2014-03-20 09:35 /home/dimDateGZip/_SUCCESS
-rw-r--r--   1 root supergroup       2266 2014-03-20 09:34 /home/dimDateGZip/part-m-00000.gz
-rw-r--r--   1 root supergroup       2461 2014-03-20 09:34 /home/dimDateGZip/part-m-00001.gz
-rw-r--r--   1 root supergroup       1905 2014-03-20 09:34 /home/dimDateGZip/part-m-00002.gz
-rw-r--r--   1 root supergroup       2814 2014-03-20 09:34 /home/dimDateGZip/part-m-00003.gz
-rw-r--r--   1 root supergroup       1546 2014-03-20 09:35 /home/dimDateGZip/part-m-00004.gz
-rw-r--r--   1 root supergroup       2804 2014-03-20 09:34 /home/dimDateGZip/part-m-00005.gz
-rw-r--r--   1 root supergroup         20 2014-03-20 09:34 /home/dimDateGZip/part-m-00006.gz
-rw-r--r--   1 root supergroup         20 2014-03-20 09:35 /home/dimDateGZip/part-m-00007.gz
-rw-r--r--   1 root supergroup         20 2014-03-20 09:35 /home/dimDateGZip/part-m-00008.gz
-rw-r--r--   1 root supergroup        535 2014-03-20 09:35 /home/dimDateGZip/part-m-00009.gz
[root@localhost ~]# hdfs dfs -ls /home/dimDate
14/03/20 09:42:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 11 items
-rw-r--r--   1 root supergroup          0 2014-03-20 09:29 /home/dimDate/_SUCCESS
-rw-r--r--   1 root supergroup      20748 2014-03-20 09:28 /home/dimDate/part-m-00000
-rw-r--r--   1 root supergroup      22248 2014-03-20 09:28 /home/dimDate/part-m-00001
-rw-r--r--   1 root supergroup      17461 2014-03-20 09:28 /home/dimDate/part-m-00002
-rw-r--r--   1 root supergroup      25573 2014-03-20 09:29 /home/dimDate/part-m-00003
-rw-r--r--   1 root supergroup      14132 2014-03-20 09:29 /home/dimDate/part-m-00004
-rw-r--r--   1 root supergroup      25693 2014-03-20 09:29 /home/dimDate/part-m-00005
-rw-r--r--   1 root supergroup          0 2014-03-20 09:29 /home/dimDate/part-m-00006
-rw-r--r--   1 root supergroup          0 2014-03-20 09:29 /home/dimDate/part-m-00007
-rw-r--r--   1 root supergroup          0 2014-03-20 09:29 /home/dimDate/part-m-00008
-rw-r--r--   1 root supergroup       3477 2014-03-20 09:29 /home/dimDate/part-m-00009

 压和没压差10倍.

 

下一步就是把table 搞进hive 打算用RCFile