hadoop测试环境sqoop使用

提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档

Sqoop看这篇文章就够了_must contain '$conditions' in where clause._SoWhat1412的博客-CSDN博客

大数据环境

C:\Windows\System32\drivers\etc

修改ip和hostname的对应关系

127.0.0.1 view-localhost # view localhost server

192.168.52.150  hadoop01  hadoop01.itcast.cn

192.168.52.151  hadoop02  hadoop02.itcast.cn

检查环境

ping 通环境测试

检查hadoop的版本

[root@hadoop01 ~]# hadoop version

Hadoop 3.0.0-cdh6.2.1

Source code repository http://github.com/cloudera/hadoop -r 7ce7dcdb76fd7f700cd7749bbc6390f76ff2abae

Compiled by jenkins on 2019-09-11T08:04Z

Compiled with protoc 2.5.0

From source with checksum b6cc25ff15ff575f159489abbee34c5

This command was run using /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/hadoop-common-3.0.0-cdh6.2.1.jar

检查hive的版本

[root@hadoop01 ~]# hive --version

WARNING: Use "yarn jar" to launch YARN applications.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Hive 2.1.1-cdh6.2.1

Subversion file:///container.redhat7/build/cdh/hive/2.1.1-cdh6.2.1/rpm/BUILD/hive-2.1.1-cdh6.2.1 -r fc0582dc4d2e2342d1eef8cb702bad68e1e2cdd2

Compiled by jenkins on Wed Sep 11 01:31:00 PDT 2019

From source with checksum 6bf22b68335469e1b010bcfecc8ea570

检查 hive服务是否启动了

[root@hadoop01 ~]# hive

WARNING: Use "yarn jar" to launch YARN applications.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/hive-common-2.1.1-cdh6.2.1.jar!/hive-log4j2.properties Async: false

WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

hive>

WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

hive>

    >

    >

    > show databases;

OK

default

test

Time taken: 4.039 seconds, Fetched: 2 row(s)

hive>

    >

检查  sqoop是否安装

[root@hadoop01 ~]# sqoop help

Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

23/09/20 13:48:09 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1

usage: sqoop COMMAND [ARGS]

Available commands:

  codegen            Generate code to interact with database records

  create-hive-table  Import a table definition into Hive

  eval               Evaluate a SQL statement and display the results

  export             Export an HDFS directory to a database table

  help               List available commands

  import             Import a table from a database to HDFS

  import-all-tables  Import tables from a database to HDFS

  import-mainframe   Import datasets from a mainframe server to HDFS

  job                Work with saved jobs

  list-databases     List available databases on a server

  list-tables        List available tables in a database

  merge              Merge results of incremental imports

  metastore          Run a standalone Sqoop metastore

  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

检查mysql数据库

[root@hadoop01 ~]# mysql -uroot -p123456

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 70

Server version: 5.7.26 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| hive               |

| hue                |

| mysql              |

| nev                |

| oozie              |

| performance_schema |

| scm                |

| sys                |

| test               |

+--------------------+

10 rows in set (0.00 sec)

sqoop list-databases --connect jdbc:mysql://192.168.52.150:3306/ --username root --password 123456

测试sqoop数据库的连接  

[root@hadoop01 ~]# sqoop list-databases --connect jdbc:mysql://192.168.52.150:3306/ --username root --password 123456

Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

23/09/20 13:54:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1

23/09/20 13:54:17 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

23/09/20 13:54:17 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

Wed Sep 20 13:54:18 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

information_schema

hive

hue

mysql

nev

oozie

performance_schema

scm

sys

test

[root@hadoop01 ~]#

练习环境部分信息

页面部分的补充

http://hadoop01:7180/cmf/home

admin/admin

http://hadoop02:8889/hue/

http://hadoop02:8889/hue/

hue/hue