路过的南国风
路过的南国风

Hadoop集群安装部署

一、概述
随着互联网的发展，各行各业的快速发展，大数据爆炸的时代，数据处理显得越来越重要了，因此学习大数据的分析也是让各位IT人员极为重要了。学习大数据的基础，就得从hadoop开始，后续要学习到的kafka、Hbase、Hive、flink等，都会多少使用到这个hadoop的相关内容，特别是HDFS和YARN两大组件功能。
后续我们是先搭建hadoop的安全分布式环境。

二、环境准备
1、服务器准备，为了体现集群安装高可用，申请了三台集群
机器名称 Ip地址 操作系统 位数 备注
hadoop01 10.10.36.20 Linux 64 Master
hadoop02 10.10.36.164 Linux 64 slaver
hadoop03 10.10.5.247 Linux 64 slaver
另外要注意hosts文件的修改。
2、SSH无密授权登录
设置ssh无密授权登录，是为了后续各类集群通信做准备。
3、Java安装准备
基于hadoop的安装，建立在Java的基础上，因此Java的安装要准备。
可以到官网下载，我准备的版本是jdk-8u181-linux-x64.tar.gz
4、Hadoop安装准备
我准备的版本是hadoop-3.2.1.tar.gz，可以到官网下载。
三、安装步骤
1、首先是配置好hosts文件，在三台机器的/etc/hosts文件编辑，添加好hosts文件

1.png

2、ssh无密授权登录配置，在hadoop01服务器，进入/root/.ssh目录，执行ssh-keygen -t rsa，回车三次


2.png

会得到id_rsa（私钥）和id_rsa.pub（公钥）两个文件，在其余两台机器都执行相同的操作，vi打开id_rsa.pub，将内容统一拷贝authorized_keys文件，包括hadoop02和hadoop03的id_rsa.pub文件内容。最后再统一将authorized_keys文件拷贝到hadoop02和hadoop03。最后可以验证ssh免密登录了。如需详细了解SSH免密登录，可以自行谷歌或百度了解。

3、检查Java的安装，如下：


1.png

由于我的机器已经安装了，也配置了jdk的环境变量，所以就不再赘述，如需了解，自行Google或百度JDK-Linux环境变量设置。
4、Hadoop安装，到了安装的重头戏了。
首先我们需要在服务器home目录下建一个子目录


2.png

将hadoop压缩包解压到/home/hadoop-ha/目录下

3.png

解压如下
4.png

接着修改环境配置文件，/etc/profile文件
export JAVA_HOME=/opt/java/jdk1.8.0_181
export PATH=PATH
export CLASSPATH=.:JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/home/hadoop-ha/hadoop-3.2.1
5.png

保存退出。
接入进入/home/hadoop-ha/hadoop-3.2.1/etc/hadoop 修改6个文件
首先是修改hadoop-env.sh，增加JAVA_HOME配置
6.png

接着修改hdfs-site.xml文件


dfs.namenode.http-address
hadoop01:50070


dfs.namenode.name.dir
file:/home/hadoop-ha/dfs/name


dfs.datanode.data.dir
file:/home/hadoop-ha/dfs/data


dfs.replication
2


dfs.webhdfs.enabled
true


dfs.permissions
false


dfs.namenode.secondary.http-adddress
hadoop01:9001


dfs.namenode.fs-limts.min-block-size
1024


接着是core-site.xml文件的配置



fs.defaultFS
hdfs://hadoop01:9000


hadoop.tmp.dir
/home/hadoop-ha/tmp


fs.default.name
hdfs://hadoop01:8020

还有修改mapred-site.xml文件的配置



mapreduce.framework.name
yarn


mapreduce.jobhistory.address
hadoop01:10020


mapreduce.jobhistory.webapp.address
hadoop01:19888


mapreduce.application.classpath
HADOOP_MAPRED_HOME/sharehadoop/mapreduce/lib/*

Yarn-site.xml文件的配置



yarn.resourcemanager.hostname
hadoop01


yarn.nodemanager.aux-services
mapreduce_shuffle


yarn.resourcemanager.webapp.address
hadoop01:8088


yarn.scheduler.maximum-allocation-mb
2048


yarn.nodemanager.vmem-check-enabled
false


yarn.nodemanager.env-whitelist
JAVA_HOME,HADOOP_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME


yarn.application.classpath
/home/hadoop-ha/hadoop-3.2.1/etc/hadoop:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/lib/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/accessors-smart-1.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/asm-5.0.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/avro-1.7.7.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/checker-qual-2.5.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-codec-1.11.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-compress-1.18.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-io-2.5.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-lang3-3.7.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-net-3.6.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/commons-text-1.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/curator-client-2.13.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/curator-framework-2.13.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/dnsjava-2.1.7.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/error_prone_annotations-2.2.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/failureaccess-1.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/hadoop-annotations-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/hadoop-auth-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/httpclient-4.5.6.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/httpcore-4.4.10.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jackson-annotations-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jackson-core-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jackson-databind-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jersey-core-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jersey-json-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jersey-server-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jersey-servlet-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-http-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-io-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-security-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-server-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-servlet-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-util-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-webapp-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jetty-xml-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jsch-0.1.54.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/json-smart-2.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-client-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-common-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-core-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-server-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerb-util-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerby-config-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerby-util-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/metrics-core-3.2.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/netty-3.10.5.Final.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/re2j-1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/snappy-java-1.0.5.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/stax2-api-3.1.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/token-provider-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib/zookeeper-3.4.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/hadoop-common-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/hadoop-common-3.2.1-tests.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/hadoop-kms-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/hadoop-nfs-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/jdiff:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/lib:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/sources:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/common/webapps:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/asm-5.0.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/avro-1.7.7.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-compress-1.18.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-io-2.5.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-net-3.6.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/commons-text-1.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/error_prone_annotations-2.2.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/gson-2.2.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/hadoop-annotations-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/hadoop-auth-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/httpclient-4.5.6.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/httpcore-4.4.10.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jackson-annotations-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jackson-core-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jackson-databind-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jettison-1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-http-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-io-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-security-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-server-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-servlet-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-util-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-util-ajax-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-webapp-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jetty-xml-9.3.24.v20180605.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/json-smart-2.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/okio-1.6.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/paranamer-2.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/re2j-1.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-3.2.1-tests.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-client-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-client-3.2.1-tests.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.1-tests.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-nfs-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.1-tests.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/jdiff:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/lib:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/sources:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/hdfs/webapps:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.1-tests.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/jdiff:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/lib:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/lib-examples:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/mapreduce/sources:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/fst-2.50.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/guice-4.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/jackson-jaxrs-base-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.9.8.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/java-util-1.9.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/jersey-client-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/json-io-2.5.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/objenesis-1.0.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-api-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-client-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-common-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-registry-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-common-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-router-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-tests-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-services-api-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-services-core-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/hadoop-yarn-submarine-3.2.1.jar:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/lib:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/sources:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/test:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/timelineservice:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/webapps:/home/hadoop-ha/hadoop-3.2.1/share/hadoop/yarn/yarn-service-examples

修改works文件，添加两个服务器主机
hadoop02
Hadoop03
创建目录


7.png

最后要配置一下/home/hadoop-ha/hadoop-3.2.1/sbin/ 目录下的start-dfs.sh、stop-dfs.sh文件
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
同时也要配置一下/home/hadoop-ha/hadoop-3.2.1/sbin/ 目录下的start-yarn.sh、stop-yarn.sh文件
YARN_RESOURCEMANAGER_USER=root
HDFS_DATANODE_USER=yarn
YARN_NODEMANAGER_USER=root

接着是将整个hadoop-ha目录文件同步到另外两台服务器。命令如下：
scp -r hadoop-ha root@hadoop02:/home/
scp -r hadoop-ha root@hadoop03:/home/

同时也同步/etc/profile文件
scp /etc/profile root@hadoop02:/etc/
scp /etc/profile root@hadoop03:/etc/

5、启动hadoop
由于hadoop01是namenode，而hadoop02和hadoop03是datanode，所以首次启动，需要对hadoop01进行初始化操作，也就是对hdfs进行格式化化。进入到/home/hadoop-ha/hadoop-3.2.1/bin目录
需要到hadoop01机器执行初始化脚本，也就是执行命令：
./hadoop namenode -format


8.png
9.png

当看到has been successfully formatted 这些提示，即为格式化成功。
最后则可以启动整体分布式服务了。进入/home/hadoop-ha/hadoop-3.2.1/sbin 目录
执行初始化脚本，也就是执行命令
./start-all.sh

10.png

使用jps命令，看到有namenode、resourcemanager、secondarynamenode进程，则说明是启动成功。
而另外两台机器，也可以jps命令查看
hadoop02


11.png

hadoop03


12.png

同时也可以登录WebUI页面登录查看，如下：
13.png

6、至此，整体的hadoop分布式环境搭建已经是完成，成功启动服务。
四、题外话
后续会逐渐开展相应的技术安装开发，都会编写相关的技术文档，欢迎拍砖纠正。

你可能感兴趣的:(Hadoop集群安装部署)

  • Python 实现图片裁剪（附代码） | Python工具 剑客阿良_ALiang
    前言本文提供将图片按照自定义尺寸进行裁剪的工具方法，一如既往的实用主义。环境依赖ffmpeg环境安装，可以参考我的另一篇文章：windowsffmpeg安装部署_阿良的博客-CSDN博客本文主要使用到的不是ffmpeg，而是ffprobe也在上面这篇文章中的zip包中。ffmpy安装：pipinstallffmpy-ihttps://pypi.douban.com/simple代码不废话了，上代码
  • openssl+keepalived安装部署 _小亦_ 项目部署keepalivedopenssl
    文章目录OpenSSL安装下载地址编译安装修改系统配置版本Keepalived安装下载地址安装遇到问题安装完成配置文件keepalived运行检查运行状态查看系统日志修改服务service重新加载systemd检查配置文件语法错误OpenSSL安装下载地址考虑到后面设备可能没法连接到外网，所以采用安装包的方式进行部署，下载地址：https://www.openssl.org/source/old/
  • Spark集群的三种模式 MelodyYN #Sparksparkhadoopbigdata
    文章目录1、Spark的由来1.1Hadoop的发展1.2MapReduce与Spark对比2、Spark内置模块3、Spark运行模式3.1Standalone模式部署配置历史服务器配置高可用运行模式3.2Yarn模式安装部署配置历史服务器运行模式4、WordCount案例1、Spark的由来定义：Hadoop主要解决，海量数据的存储和海量数据的分析计算。Spark是一种基于内存的快速、通用、可
  • Linux学习-Ansible（一） 丢爸 Linuxlinux学习ansible
    环境-Rocky-Linux8.6安装部署Ansible#安装ansible[root@harboransible]#dnfinstall-yansible-core#查看安装信息[root@harboransible]#ansible-doc--versionansible-doc[core2.12.2]configfile=/root/ansible/ansible.cfgconfigured
  • DMDPC安装部署 丷月亮是指路牌 数据库
    1.DMDPC架构DMDPC架构由计划生成节点SP、数据存储节点BP和元数据服务器节点MP三部分组成：1、SP对外提供分布式数据库服务，用户可以登录到任意一个SP节点，获得完整的数据库服务；2、BP负责存储数据，执行SP的调度指令并将执行结果返回给SP；3、MP负责存储元数据并向SP、BP提供元数据服务。本文是在1台Linux虚拟机上通过创建四个实例的方式进行集群搭建.2.DMDPC集群规划DMD
  • 银河麒麟高级服务器操作系统V10SP2（X86）PXE服务端部署以及测试 熊08 服务器运维linuxkylin
    银河麒麟高级服务器操作系统V10SP2（X86）PXE服务端部署以及测试一系统环境1.1服务端/客户端系统版本1.2服务端环境信息二组件安装部署2.1安装软件包2.1.1uefi配置2.1.2legacy配置2.2dhcp服务配置2.2.1uefi配置2.2.2legacy配置2.3xinetd服务配置2.4http服务配置2.4.1uefi配置2.4.2legacy配置2.5grub配置2.5.
  • 阿里云服务器Zookeeper单机安装部署（单机伪集群稍后更新） 不减30斤不改名_TC
    首先服务器系统版本配置如下：40e2c9e8ed734139ac1f99197a01dad8.pngzookeeper下载链接：http://mirror.bit.edu.cn/apache/zookeeper/38f2d0f359d7426faf44a6b8047f4a94.pngzookeeper版本很重要，我下载的是3.4.14版本，至于为什么不用新的，后面会讲到。下载后解压放置路径：028
  • 【Docker】MongoDB 容器化部署 轻云UC 运维部署mongodbdocker数据库
    MongoDBdocker容器化部署MongoDB标准软件基于BitnamiMongoDB构建。当前版本未7.0.5你可以通过轻云UC部署工具直接安装部署，也可以手动按如下文档操作配置文件地址:https://gitee.com/qingplus/qingcloud-platformQinghubStudio在线体验初始化一个新实例使用Docker容器网络，应用程序容器可以轻松访问容器内运行的Mo
  • 01-Flink安装部署及入门案例（仅供学习），音视频时代你还不会NDK开发 小猪佩琪962 2024年程序员学习flink学习大数据
    先自我介绍一下，小编浙江大学毕业，去过华为、字节跳动等大厂，目前阿里P7深知大多数程序员，想要提升技能，往往是自己摸索成长，但自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！因此收集整理了一份《2024年最新大数据全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友。既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵
  • 解决Windows环境下hadoop集群的运行_window运行hadoop,unknown hadoop01(4) 2401_84160087 大数据面试学习
    网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。需要这份系统化资料的朋友，可以戳这里获取一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！org.apache.hadoophadoop-com
  • 解决Windows环境下hadoop集群的运行_window运行hadoop,unknown hadoop01(3) 2401_84160087 大数据面试学习
    网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。需要这份系统化资料的朋友，可以戳这里获取一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！xmlns:xsi="http://www.w3.or
  • hive搭建 -----内嵌模式和本地模式 lzhlizihang hivehadoop
    文章目录一、内嵌模式（使用较少）1、上传、解压、重命名2、配置环境变量3、配置conf下的hive-env.sh4、修改conf下的hive-site.xml5、启动hadoop集群6、给hdfs创建文件夹7、修改hive-site.xml中的非法字符8、初始化元数据9、测试是否成功10、内嵌模式的缺点二、本地模式（最常用）1、检查mysql是否正常2、上传、解压、重命名3、配置环境变量4、修改c
  • 如何在虚拟机中安装部署K8S？ 桃林春风一杯酒 kubernetesjavadocker
    教程参考：centos7安装k8s1.28版本，基于科学-CSDN博客环境准备：准备三台机器，都做以下操作，或者只准备一个机器，最后再克隆两台。yum：换源，这是阿里云的源sudowget-O/etc/yum.repos.d/CentOS-Base.repohttp://mirrors.aliyun.com/repo/Centos-7.repo然后清缓存，建立元数据yumcleanallyumma
  • Linux - Linux安装部署Maven以及环境变量配置 MinggeQingchun LinuxMavenlinuxmaven运维
    1、下载Maven：前往官网下载最新版的Maven：Maven–DownloadApacheMaven2、解压Maven：将下载的压缩包解压到你希望安装Maven的目录，例如：tar-zxvfapache-maven-3.8.6-bin.tar.gz-C/opt/maven3、配置环境变量：编辑.bashrc或.bash_profile文件，添加以下内容：exportM2_HOME=/opt/ma
  • 史上最全LVS+keepalived+nginx基于DR模式高可用安装部署 徐兴强的博客 负载均衡服务器linux
    史上最全LVS+keepalived+nginx基于DR模式高可用安装部署lvs+keepAlived→效率最高的负载均衡在大型网站中一般服务端会做集群，同时利用负载均衡器做负载均衡。这样有利于将大量的请求分散到各个服务器上，提升网站的响应速度。当然为了解决单点故障的问题，还会做热备份方案。这里演示利用LVS做负载均衡器，同时利用Keepalived保证其高可用，基于LVS的DR模式构建Nginx
  • helm2和helm3的区别及helm2的安装使用 Rami
    1.从Helm2到Helm3helm是一个k8s应用包管理工具，负责云原生应用的安装部署和生命周期管理2.Helm2两个主要的组件Tiller：helm的服务端，部署在k8s里面的一个pod，通常在kube-system这个名称空间里，主要负责部署helmcharts，管理release，跟k8sAPIserver通信。helmClient：helm的客户端，主要负责从公有或者私有charts仓库
  • Aarch64架构OpenEuler服务器安装部署.NetCore站点 東云dove .netcorelinux服务器visualstudio
    下载对应的离线包（SDK或者runtime）在OpenEuler中找到/usr/share/并创建dotnet目录，并将下载的离线包解压添加环境变量exportPATH=$PATH:/usr/share/dotnetsource~/.bashrcdotnet--info将.NetCore站点发布为linux-arm64（旧版本的VS没有这个选项，我用的是VS2022发布的）进入对应的文件夹，运行d
  • 隐语课程 SecretFlow的安装部署 皓月雪 学习笔记
    SecretFlow安装：1.先开启电脑上的虚拟机功能，以管理员身份打开PowerShell并运行：dism.exe/online/enable-feature/featurename:VirtualMachinePlatform/all/norestart2.在Windows上安装WSL2，安装WSL|MicrosoftLearn3.将WSL2设置为默认版本，以管理员身份打开PowerShell
  • Docker无介绍快使用，docker数据卷Volumes挂载的使用（三） 时间是一种毒药
    问题背景上篇介绍了Docker指令，本章介绍一下数据卷VolumesDocker无介绍快使用，安装部署hello测试（一）Docker无介绍快使用，docker指令使用（二）Docker无介绍快使用，docker数据卷Volumes挂载的使用（三）Docker数据卷Volumes概述：数据卷是宿主机中的一个目录或文件，当容器目录和数据卷目录绑定后，对方的修改会立即同步一个数据卷可以被多个容器同时挂
  • MySQL gh-ost DDL 变更工具 Bing@DBA MySQLmysql数据库
    文章目录1.MDL锁介绍2.变更工具3.gh-ost原理解析4.安装部署5.操作演示5.1.重点参数介绍5.2.执行变更5.3.动态控制6.风险提示1.MDL锁介绍MySQL的锁可以分为四类：MDL锁、表锁、行锁、GAP锁，其中除了MDL锁是在Server层加的之外，其它三种都是在InnoDB层加的。下面主要介绍一下：MDL元数据锁，主要作用就是维护DDL过程中数据的安全性&正确性。当对一个表进行
  • 【网络架构】ELK 云计算稿手 网络架构架构elk
    目录一、ELK介绍1.1ELK是什么1.2Elasticsearch1.3Logstash1.4Kibana1.5为什么使用ELK二、ELK原理2.1工作原理2.2ELK的应用架构图三、ELK的安装部署3.1环境部署3.2前期环境3.3安装Elasticsearch3.3.1修改配置3.3.2查看节点信息3.4安装Logstash3.4.1安装服务3.4.2使用Logstash3.4.3对接ela
  • powerdns mysql_PowerDNS安装部署 泰坦V powerdnsmysql
    centos安装epel检验epel[[email protected]]#lsCentOS-Base.repoCentOS-fasttrack.repoCentOS-Vault.repoCentOS-CR.repoCentOS-Media.repoepel.repoCentOS-Debuginfo.repoCentOS-Sources.repoepel-testing.repo安装
  • WebService环境安装+编写+发布部署+调用 时光书签 c#专题c#webserviceiis
    内容目录：1.WebService环境安装2.WebService编写3.WebService的发布及部署4.WebService调用一、WebService环境安装部署IIS环境。win7中打开控制面板--->程序和功能--->打开或关闭Window功能，按以下安装好，就部署好IIS环境了。二、WebService编写1.我使用的是VS2019.首先，在解决方案中，右击解决方案->添加->新建项
  • Python 全栈系列266 Kafka服务的Docker搭建 yukai08008 全栈pythonkafkadocker
    说明在大量数据处理任务下的缓存与分发这个算是来自顾同学的助攻+1，我有点java绝缘体的体质，碰到和java相关的安装部署总会碰到点奇怪的问题，不过现在已经搞定了。测试也接近了kafka官方标称的性能。考虑到网络、消息的大小等因素，可以简单认为kafka的速度是10万/秒级的。本次文章的目的是：1搭建一个平时工作中常用的队列服务2方便自己或者其他同事再次搭建内容1搭建过程共要搭建两个服务：zook
  • 国产海量数据库安装部署文档 人生不过大梦一场 db数据库windowslinux
    本系统为麒麟V10系统,安装用户为普通用户给予sudo权限，主机内存16G（8G安装没装起来）#默认端口5432对外端口数据库对外提供访问5433对内端口Basebackup与复制流协议系统环境设置#关闭防火墙（二者执行一个）sudosystemctlstopfirewalld.service&&sudosystemctldisablefirewalld&&sudosystemctlstatusf
  • 数据仓库之【商品订单数据数仓】10：数据可视化工具：Zeppelin安装部署、Zeppelin使用 做一个有趣的人Zz hadoophive数据仓库hivebigdata
    一、数据可视化数据可视化这块不是项目的重点，不过为了让大家能有一个更加直观的感受，我们可以选择一些现成的数据可视化工具实现。咱们前面分析过，想要查询hive中的数据可以使用hue，不过hue无法自动生成图表。所以我们可以考虑使用Zeppelin针对一些复杂的图表，可以选择定制开发，使用echarts、finebi组件实现。二、Zeppelin安装部署注意：不要使用Zeppelin0.8.2版本，这
  • K8S安装部署 一心只为学 kuberneteslinux容器
    一、准备工作在vmware中安装，创建三台centos服务器主机ip配置master192.168.42.1884核，6g内存，40g硬盘node1192.168.42.1892核，2g内存，20g硬盘node2192.168.42.1902核，2g内存，20g硬盘二、环境配置1、修改hosts配置（所有节点执行）vim/etc/hosts192.168.42.188master192.168.4
  • K8S笔记：Kubernetes安装部署(CentOS7) arksea devopskubernetesk8s运维
    目录文章目录目录一、使用minikube启动minikube的LoadBalancer支持指定minikube自带docker的docker0网桥网段minikube使用本地image二、使用kubeadm手工安装1、安装前准备2、允许iptables检查桥接流量3、安装kubeletkubeadmkubectl4、kubelet使用cgroup驱动5、确认主机名/域名可被访问6、关闭Swap或忽
  • centos安装k8s集群（kubeadm方式） MssGuo kubernetescentoskubernetesLinux运维
    目录前言生产环境部署k8s集群的两种方式服务器初始化、环境准备使用kubeadm安装k8s（本篇讲解使用kubeadm安装k8s）后记-k8s使用外部etcd集群前言环境：centos7.9docker-ce-20.10.9kubernetes-versionv1.22.17本篇来讲解如何在centos下安装部署k8s集群生产环境部署k8s集群的两种方式kubeadmkubeadm是一个工具，提供
  • MySQL中如何用全库备份恢复单张表 As before@ mysql数据库运维
    一、概述模拟场景：下面模拟逻辑备份与物理备份两种方式恢复单表方法，利用完整备份+binlog恢复单表，某天某张表意外删除，如何从全备中恢复单表？二、安装部署2.1Mysql安装（我的模拟环境数据版本为：5.7.36，此处省略相关操作）2.2xtrabackup-2.4.24安装（物理备份）官网地址：https://www.percona.com/downloads/Percona-XtraBack
  • ASM系列四 利用Method 组件动态注入方法逻辑 lijingyao8206 字节码技术jvmAOP动态代理ASM
            这篇继续结合例子来深入了解下Method组件动态变更方法字节码的实现。通过前面一篇，知道ClassVisitor 的visitMethod()方法可以返回一个MethodVisitor的实例。那么我们也基本可以知道，同ClassVisitor改变类成员一样，MethodVIsistor如果需要改变方法成员，注入逻辑，也可以
  • java编程思想 --内部类 百合不是茶 java内部类匿名内部类
    内部类;了解外部类 并能与之通信 内部类写出来的代码更加整洁与优雅   1,内部类的创建  内部类是创建在类中的 package com.wj.InsideClass; /* * 内部类的创建 */ public class CreateInsideClass { public CreateInsideClass(
  • web.xml报错 crabdave web.xml
    web.xml报错   The content of element type "web-app" must match "(icon?,display-  name?,description?,distributable?,context-param*,filter*,filter-mapping*,listener*,servlet*,s
  • 泛型类的自定义 麦田的设计者 javaandroid泛型
       为什么要定义泛型类，当类中要操作的引用数据类型不确定的时候。 采用泛型类，完成扩展。   例如有一个学生类     Student{ Student(){ System.out.println("I'm a student....."); } }  有一个老师类  
  • CSS清除浮动的4中方法 IT独行者 JavaScriptUIcss
    清除浮动这个问题，做前端的应该再熟悉不过了，咱是个新人，所以还是记个笔记，做个积累，努力学习向大神靠近。CSS清除浮动的方法网上一搜，大概有N多种，用过几种，说下个人感受。 1、结尾处加空div标签 clear:both 1 2 3 4 .div 1 { background : #000080 ; border : 1px   s
  • Cygwin使用windows的jdk 配置方法 _wy_ jdkwindowscygwin
    1.[vim /etc/profile]    JAVA_HOME="/cgydrive/d/Java/jdk1.6.0_43"  (windows下jdk路径为D:\Java\jdk1.6.0_43)    PATH="$JAVA_HOME/bin:${PATH}"    CLAS
  • linux下安装maven 无量 mavenlinux安装
    Linux下安装maven(转) 1.首先到Maven官网 下载安装文件，目前最新版本为3.0.3，下载文件为 apache-maven-3.0.3-bin.tar.gz，下载可以使用wget命令； 2.进入下载文件夹，找到下载的文件，运行如下命令解压 tar -xvf  apache-maven-2.2.1-bin.tar.gz 解压后的文件夹
  • tomcat的https 配置,syslog-ng配置 aichenglong tomcathttp跳转到httpssyslong-ng配置syslog配置
    1) tomcat配置https,以及http自动跳转到https的配置     1)TOMCAT_HOME目录下生成密钥(keytool是jdk中的命令)      keytool -genkey -alias tomcat -keyalg RSA -keypass changeit -storepass changeit
  • 关于领号活动总结 alafqq 活动
    关于某彩票活动的总结 具体需求，每个用户进活动页面，领取一个号码，1000中的一个； 活动要求 1，随机性，一定要有随机性； 2，最少中奖概率，如果注数为3200注，则最多中4注 3，效率问题，（不能每个人来都产生一个随机数，这样效率不高）； 4，支持断电（仍然从下一个开始），重启服务；（存数据库有点大材小用，因此不能存放在数据库） 解决方案 1，事先产生随机数1000个，并打
  • java数据结构 冒泡排序的遍历与排序 百合不是茶 java
    java的冒泡排序是一种简单的排序规则   冒泡排序的原理：           比较两个相邻的数，首先将最大的排在第一个，第二次比较第二个 ，此后一样；         针对所有的元素重复以上的步骤，除了最后一个     例题；将int array[]
  • JS检查输入框输入的是否是数字的一种校验方法 bijian1013 js
    如下是JS检查输入框输入的是否是数字的一种校验方法： <form method=post target="_blank"> 数字：<input type="text" name=num onkeypress="checkNum(this.form)"><br> </form>
  • Test注解的两个属性：expected和timeout bijian1013 javaJUnitexpectedtimeout
    JUnit4：Test文档中的解释： 　　The Test annotation supports two optional parameters. 　　The first, expected, declares that a test method should throw an exception. 　　If it doesn't throw an exception or if it
  • [Gson二]继承关系的POJO的反序列化 bit1129 POJO
    父类     package inheritance.test2; import java.util.Map; public class Model { private String field1; private String field2; private Map<String, String> infoMap
  • 【Spark八十四】Spark零碎知识点记录 bit1129 spark
    1. ShuffleMapTask的shuffle数据在什么地方记录到MapOutputTracker中的 ShuffleMapTask的runTask方法负责写数据到shuffle map文件中。当任务执行完成成功，DAGScheduler会收到通知，在DAGScheduler的handleTaskCompletion方法中完成记录到MapOutputTracker中  
  • WAS各种脚本作用大全 ronin47 WAS 脚本
    　　　http://www.ibm.com/developerworks/cn/websphere/library/samples/SampleScripts.html 　　　无意中，在WAS官网上发现的各种脚本作用，感觉很有作用，先与各位分享一下 　　　 获取下载 这些示例 jacl 和 Jython 脚本可用于在 WebSphere Application Server 的不同版本中自
  • java-12.求 1+2+3+..n不能使用乘除法、 for 、 while 、 if 、 else 、 switch 、 case 等关键字以及条件判断语句 bylijinnan switch
    借鉴网上的思路，用java实现： public class NoIfWhile { /** * @param args * * find x=1+2+3+....n */ public static void main(String[] args) { int n=10; int re=find(n); System.o
  • Netty源码学习-ObjectEncoder和ObjectDecoder bylijinnan javanetty
    Netty中传递对象的思路很直观： Netty中数据的传递是基于ChannelBuffer（也就是byte[]）； 那把对象序列化为字节流，就可以在Netty中传递对象了 相应的从ChannelBuffer恢复对象，就是反序列化的过程 Netty已经封装好ObjectEncoder和ObjectDecoder 先看ObjectEncoder ObjectEncoder是往外发送
  • spring 定时任务中cronExpression表达式含义 chicony cronExpression
    一个cron表达式有6个必选的元素和一个可选的元素，各个元素之间是以空格分隔的，从左至右，这些元素的含义如下表所示： 代表含义            是否必须 允许的取值范围         &nb
  • Nutz配置Jndi ctrain JNDI
    1、使用JNDI获取指定资源： var ioc = { dao : { type :"org.nutz.dao.impl.NutDao", args : [ {jndi :"jdbc/dataSource"} ] } } 以上方法,仅需要在容器中配置好数据源,注入到NutDao即可.
  • 解决 /bin/sh^M: bad interpreter: No such file or directory daizj shell
    在Linux中执行.sh脚本，异常/bin/sh^M: bad interpreter: No such file or directory。   分析：这是不同系统编码格式引起的：在windows系统中编辑的.sh文件可能有不可见字符，所以在Linux系统下执行会报以上异常信息。 解决： 1）在windows下转换： 利用一些编辑器如UltraEdit或EditPlus等工具
  • [转]for 循环为何可恨？ dcj3sjt126com 程序员读书
      Java的闭包(Closure)特征最近成为了一个热门话题。 一些精英正在起草一份议案，要在Java将来的版本中加入闭包特征。 然而，提议中的闭包语法以及语言上的这种扩充受到了众多Java程序员的猛烈抨击。 不久前，出版过数十本编程书籍的大作家Elliotte Rusty Harold发表了对Java中闭包的价值的质疑。 尤其是他问道“for 循环为何可恨？”[http://ju
  • Android实用小技巧 dcj3sjt126com android
    1、去掉所有Activity界面的标题栏 　　修改AndroidManifest.xml 　　在application 标签中添加android:theme="@android:style/Theme.NoTitleBar"   2、去掉所有Activity界面的TitleBar 和StatusBar  　　修改AndroidManifes
  • Oracle 复习笔记之序列 eksliang Oracle 序列sequenceOracle sequence
    转载请出自出处：http://eksliang.iteye.com/blog/2098859 1.序列的作用 序列是用于生成唯一、连续序号的对象 一般用序列来充当数据库表的主键值 2.创建序列语法如下：   create sequence s_emp start with 1 --开始值 increment by 1 --増长值 maxval
  • 有“品”的程序员 gongmeitao 工作
    完美程序员的10种品质　　   完美程序员的每种品质都有一个范围，这个范围取决于具体的问题和背景。没有能解决所有问题的   完美程序员（至少在我们这个星球上），并且对于特定问题，完美程序员应该具有以下品质： 　　1. 才智非凡- 能够理解问题、能够用清晰可读的代码翻译并表达想法、善于分析并且逻辑思维能力强   （范围：用简单方式解决复杂问题）　　
  • 使用KeleyiSQLHelper类进行分页查询 hvt sql.netC#asp.nethovertree
    本文适用于sql server单主键表或者视图进行分页查询，支持多字段排序。KeleyiSQLHelper类的最新代码请到http://hovertree.codeplex.com/SourceControl/latest下载整个解决方案源代码查看。或者直接在线查看类的代码：http://hovertree.codeplex.com/SourceControl/latest#HoverTree.D
  • SVG 教程 （三）圆形，椭圆，直线 天梯梦 svg
    SVG <circle> SVG 圆形 - <circle> <circle> 标签可用来创建一个圆： 下面是SVG代码： <svg xmlns="http://www.w3.org/2000/svg" version="1.1"> <circle cx="100" c
  • 链表栈 luyulong java数据结构
    public class Node { private Object object; private Node next; public Node() { this.next = null; this.object = null; } public Object getObject() { return object; } public
  • 基础数据结构和算法十：2-3 search tree sunwinner Algorithm2-3 search tree
      Binary search tree works well for a wide variety of applications, but they have poor worst-case performance. Now we introduce a type of binary search tree where costs are guaranteed to be loga
  • spring配置定时任务 stunizhengjia springtimer
    最近因工作的需要，用到了spring的定时任务的功能,觉得spring还是很智能化的,只需要配置一下配置文件就可以了,在此记录一下，以便以后用到：     //------------------------定时任务调用的方法------------------------------ /** * 存储过程定时器 */ publi
  • ITeye 8月技术图书有奖试读获奖名单公布 ITeye管理员 活动
    ITeye携手博文视点举办的8月技术图书有奖试读活动已圆满结束，非常感谢广大用户对本次活动的关注与参与。 8月试读活动回顾： http://webmaster.iteye.com/blog/2102830 本次技术图书试读活动的优秀奖获奖名单及相应作品如下（优秀文章有很多，但名额有限，没获奖并不代表不优秀）： 《跨终端Web》 gleams：http
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他