–append 将数据追加到hdfs中已经存在的dataset中。使用该参数,sqoop将把数据先导入到一个临时目录中,
然后重新给文件命名到一个正式的目录中,以避免和该目录中已存在的文件重名。
–as-avrodatafile 将数据导入到一个Avro数据文件中
–as-sequencefile 将数据导入到一个sequence文件中
–as-textfile 将数据导入到一个普通文本文件中,生成该文本文件后,可以在hive中通过sql语句查询出结果。
–boundary-query 边界查询,也就是在导入前先通过SQL查询得到一个结果集,然后导入的数据就是该结果集内的数据,
格式如:–boundary-query ‘select id,creationdate from person where id = 3’,表示导入的数据为id=3的记录,
或者select min(), max() from
,注意查询的字段中不能有数据类型为字符串的字段,
否则会报错:java.sql.SQLException: Invalid value for getLong()目前问题原因还未知
–columns
执行报错:
19/03/05 17:48:25 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
19/03/05 17:48:25 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
... 12 more
解决办法:
1、将sqoop安装目录下的conf目录中的sqoop-env-template.sh复制为sqoop-env.sh
并且指定
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/apps/hadoop-2.8.0
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/apps/hadoop-2.8.0
#set the path to where bin/hbase is available
export HBASE_HOME=/apps/hbase-1.2.6
#Set the path to where bin/hive is available
export HIVE_HOME=/apps/hive-1.2.1
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/apps/zookeeper-3.4.10
2、解决办法
在/etc/profile中添加:
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*:$HIVE_CONF_DIR
记得source /etc/profile
在~/.bash_profile中添加:
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*:$HIVE_CONF_DIR
记得source ~/.bash_profile
mysql数据样例:
/*
Navicat Premium Data Transfer
Source Server : localhost-mysql
Source Server Type : MySQL
Source Server Version : 50722
Source Host : localhost:3306
Source Schema : hadoop_dw
Target Server Type : MySQL
Target Server Version : 50722
File Encoding : 65001
Date: 04/03/2019 09:24:33
*/
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
-- ----------------------------
-- Table structure for sx_hospital
-- ----------------------------
DROP TABLE IF EXISTS `sx_hospital`;
CREATE TABLE `sx_hospital` (
`name` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`latlng` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`province` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`city` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`area` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`address` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
PRIMARY KEY (`latlng`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of sx_hospital
-- ----------------------------
INSERT INTO `sx_hospital` VALUES ('济民医院', '110.31956,34.646432', '山西省', '运城市', '芮城县', '风陵渡镇');
INSERT INTO `sx_hospital` VALUES ('蒲州医院', '110.335746,34.837638', '山西省', '运城市', '永济市', '山西省运城市永济市张风线蒲州镇中心卫生院');
INSERT INTO `sx_hospital` VALUES ('健民医院', '110.435347,34.886854', '山西省', '运城市', '永济市', '涑水西街283号');
INSERT INTO `sx_hospital` VALUES ('永济黄河中医院', '110.449017,34.896502', '山西省', '运城市', '永济市', '舜都大道');
INSERT INTO `sx_hospital` VALUES ('永济市忠愍医院', '110.450128,34.894908', '山西省', '运城市', '永济市', '山西省运城市永济市舜都大道');
INSERT INTO `sx_hospital` VALUES ('康宁医院', '110.455258,34.89246', '山西省', '运城市', '永济市', '828县道附近');
INSERT INTO `sx_hospital` VALUES ('永济市人民医院', '110.458271,34.868693', '山西省', '运城市', '永济市', '山西省运城市永济市银杏东街9号');
INSERT INTO `sx_hospital` VALUES ('万荣县第二人民医院', '110.554863,35.362433', '山西省', '运城市', '万荣县', '荣河镇');
INSERT INTO `sx_hospital` VALUES ('临猗县第二人民医院', '110.560605,35.10071', '山西省', '运城市', '临猗县', '临晋镇西街26号');
INSERT INTO `sx_hospital` VALUES ('山西省芮城县大王医院', '110.564027,34.693207', '山西省', '运城市', '芮城县', '山西省运城市芮城县新兴曹风线大王镇中心卫生院');
INSERT INTO `sx_hospital` VALUES ('万荣福祯医院', '110.566621,35.360069', '山西省', '运城市', '万荣县', '福祯创伤骨科医院附近');
SET FOREIGN_KEY_CHECKS = 1;
原题链接:#137 Single Number II
要求:
给定一个整型数组,其中除了一个元素之外,每个元素都出现三次。找出这个元素
注意:算法的时间复杂度应为O(n),最好不使用额外的内存空间
难度:中等
分析:
与#136类似,都是考察位运算。不过出现两次的可以使用异或运算的特性 n XOR n = 0, n XOR 0 = n,即某一
A message containing letters from A-Z is being encoded to numbers using the following mapping:
'A' -> 1
'B' -> 2
...
'Z' -> 26
Given an encoded message containing digits, det