迷途思凡
迷途思凡

Spark学习-事件日志EventLog

事件日志EventLog

  • 背景
  • 系统结构
  • Spark Event
    • 介绍
    • 事件类型
    • 事件内容
    • 事件日志
    • 事件分析

背景

    最近工作需要使用Flink对Spark的应用事件日志进行处理，帮助发现Spark应用中常见异常问题，为用户提供方便快捷的常见问题排查，在此作一下简单的记录和学习。

系统结构

Spark学习-事件日志EventLog_第1张图片

    Spark开启Thrift Server进行Spark SQL查询，Spark的Jobhistory服务对应用的事件进行记录和保存，利用FileBeat对历史事件日志传送到Kafka Topic中，最后使用Flink对Topic中的事件日志进行处理，并输出到ElasticSearch作结果展示。

Spark Event

介绍

    Spark Event是Spark为Spark App定义的运行事件，用于展示一个Spark App完整的生命周期，Spark History Server服务可将App的事件日志存储到Hdfs中(即Event Log，JSON文件格式),一般来说一个Spark Job对应一个事件日志，通过分析这些事件日志，我们可以发现一些应用中的错误和待提升的点，从而进一步去优化应用性能。

事件类型

    Spark事件类型可分为如下类型，具体可参考Spark源码SparkListener.class。
Spark学习-事件日志EventLog_第2张图片
    Spark的运行过程正是由上图中的多种不同事件组合而成，事件归纳一下主要分为Spark App运行事件、资源变更事件、环境变更事件和其他事件，图中相同颜色代表属于同一过程的不同状态的事件。

事件内容

事件名称 事件描述 事件内容
SparkListenerApplicationStart App开始事件 App Name:App名称
App ID:App的主键
Timestamp: 事件触发时间
User:提交App的用户
SparkListenerApplicationEnd App结束事件 Timestamp:事件触发时间
SparkListenerJobStart Job开始事件 Job ID:Job的主键
Submission Time:Job提交时间
Stage Infos: Job中所有stage的详细信息
Stage IDs:Job中所有stage的ID列表
Properties:Job的配置信息
SparkListenerJobEnd Job结束事件 Job ID:Job的主键
Completion Time:Job完成时间
Job Result: Job的执行结果信息
SparkListenerStageSubmitted Stage提交事件 Stage Info:当前提交Stage的详细信息
Properties:Stage的配置信息
SparkListenerStageCompleted Stage完成事件 Stage Info:当前完成Stage的详细信息
SparkListenerTaskStart Task启动事件 Stage ID:任务所在Stage的主键
Stage Attempt ID:上次尝试提交Stage的主键
Task Info:Task任务的详细信息
SparkListenerTaskGettingResult Task下载任务结果事件 Task Info:Task执行的详细信息
SparkListenerTaskEnd Task任务结束事件 Stage ID:任务所在Stage的主键
Stage Attempt ID:上次尝试提交Stage的主键
Task Type:Task的类型
Task End Reason:Task结束的原因
Task Info:Task任务的详细信息
Task Executor Metrics:执行Task的Executor的JVM指标信息
Task Metrics:Task运行的指标信息
SparkListenerEnvironmentUpdate 环境更新事件 JVM Information:Java和Scala的版本信息
Spark Properties:App的Spark参数配置信息
Hadoop Properties:App的Yarn参数配置信息
System Properties:提交App的系统环境配置信息
Classpath Entries:环境变量信息
SparkListenerBlockManagerAdded 块管理器添加事件 Block Manager ID:块管理器的主键信息
Maximum Memory:最大内存值
Timestamp:事件触发时间
Maximum Onheap Memory:最大堆内内存值
Maximum Offheap Memory:最大堆外内存值
SparkListenerBlockManagerRemoved 块管理器移除事件 Block Manager ID:块管理器的主键信息
Timestamp:事件触发时间
SparkListenerExecutorMetricsUpdate Task指标更新事件 Executor ID:Executor的主键
Metrics Updated:Task更新的指标值
SparkListenerExecutorAdded Executor注册事件 Executor ID:Executor的主键
Executor Info:Executor的详细信息
Timestamp:事件触发时间
SparkListenerExecutorRemoved Executor移除事件 Executor ID:Executor的主键
Timestamp:事件触发时间
Removed Reason:Executor移除原因
SparkListenerBlockUpdated block更新事件 Block Updated Info:Block更新的详细信息
SparkListenerUnpersistRDD 取消 RDD缓存事件 RDD ID:RDD的主键
SparkListenerExecutorBlacklisted Executor黑名单事件 time:事件触发时间
executorId:Executor的主键
taskFailures:task失败次数
SparkListenerExecutorBlacklistedForStage 针对Stage的Executor黑名单事件 time:事件触发时间
executorId:Executor的主键
taskFailures:task失败次数
stageId:Stage的主键
stageAttemptId:上次尝试提交Stage的主键
SparkListenerNodeBlacklisted Node黑名单事件 time:事件触发时间
hostId:节点的主键
executorFailures:executor的失败次数
SparkListenerNodeBlacklistedForStage 针对Stage的Node黑名单事件 time:事件触发时间
executorId:Executor的主键
executorFailures:executor的失败次数
stageId:Stage的主键
stageAttemptId:上次尝试提交Stage的主键
SparkListenerExecutorUnblacklisted Executor黑名单移除事件 time:事件触发时间
executorId:Executor的主键
SparkListenerNodeUnblacklisted Node黑名单移除事件 time:事件触发时间
hostId:节点的主键

    表格事件内容列中，红色字段是会在事件日志中输出的JSON字段，蓝色字段是不会在事件日志中输出的后台字段。

事件日志

Spark Pi运行的事件日志如下所示：

{"Event":"SparkListenerLogStart","Spark Version":"3.0.1"}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"driver","Host":"master","Port":43711},"Maximum Memory":97832140,"Timestamp":1603121632911,"Maximum Onheap Memory":97832140,"Maximum Offheap Memory":0}
{"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java Home":"/home/java/jre","Java Version":"1.8.0_162 (Oracle Corporation)","Scala Version":"version 2.12.10"},"Spark Properties":{"spark.serializer":"org.apache.spark.serializer.KryoSerializer","spark.driver.host":"master","spark.eventLog.enabled":"true","spark.driver.port":"46515","spark.jars":"file:/home/spark-3.0.1/examples/jars/spark-examples_2.12-3.0.1.jar","spark.app.name":"Spark Pi","spark.scheduler.mode":"FIFO","spark.driver.memory":"512m","spark.submit.pyFiles":"","spark.executor.id":"driver","spark.submit.deployMode":"client","spark.master":"spark://master:7077","spark.executor.memory":"512m","spark.eventLog.dir":"hdfs://master:9000/spark_event","spark.executor.cores":"1","spark.app.id":"app-20201019233352-0002"},"Hadoop Properties":{"hadoop.service.shutdown.timeout":"30s","yarn.resourcemanager.amlauncher.thread-count":"50","yarn.sharedcache.enabled":"false","fs.s3a.connection.maximum":"15","yarn.nodemanager.numa-awareness.numactl.cmd":"/usr/bin/numactl","fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem","yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms":"1000","yarn.timeline-service.timeline-client.number-of-async-entities-to-merge":"10","hadoop.proxyuser.root.hosts":"*","hadoop.security.kms.client.timeout":"60","hadoop.http.authentication.kerberos.principal":"HTTP/_HOST@LOCALHOST","mapreduce.jobhistory.loadedjob.tasks.max":"-1","mapreduce.framework.name":"yarn","yarn.sharedcache.uploader.server.thread-count":"50","yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern":"^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$","tfile.fs.output.buffer.size":"262144","yarn.app.mapreduce.am.job.task.listener.thread-count":"30","hadoop.security.groups.cache.background.reload.threads":"3","yarn.resourcemanager.webapp.cross-origin.enabled":"false","fs.AbstractFileSystem.ftp.impl":"org.apache.hadoop.fs.ftp.FtpFs","hadoop.registry.secure":"false","hadoop.shell.safely.delete.limit.num.files":"100","dfs.bytes-per-checksum":"512","mapreduce.job.acl-view-job":" ","fs.s3a.s3guard.ddb.background.sleep":"25ms","fs.s3a.retry.limit":"${fs.s3a.attempts.maximum}","mapreduce.jobhistory.loadedjobs.cache.size":"5","fs.s3a.s3guard.ddb.table.create":"false","yarn.nodemanager.amrmproxy.enabled":"false","yarn.timeline-service.entity-group-fs-store.with-user-dir":"false","mapreduce.input.fileinputformat.split.minsize":"0","yarn.resourcemanager.container.liveness-monitor.interval-ms":"600000","yarn.resourcemanager.client.thread-count":"50","io.seqfile.compress.blocksize":"1000000","yarn.sharedcache.checksum.algo.impl":"org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl","yarn.nodemanager.amrmproxy.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor","dfs.datanode.data.dir":"/data/hadoop/hdfs/data","dfs.replication":"3","yarn.timeline-service.entity-group-fs-store.leveldb-cache-read-cache-size":"10485760","mapreduce.reduce.shuffle.fetch.retry.interval-ms":"1000","mapreduce.task.profile.maps":"0-2","yarn.scheduler.include-port-in-node-name":"false","yarn.nodemanager.admin-env":"MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX","yarn.resourcemanager.node-removal-untracked.timeout-ms":"60000","mapreduce.am.max-attempts":"2","hadoop.security.kms.client.failover.sleep.base.millis":"100","mapreduce.jobhistory.webapp.https.address":"0.0.0.0:19890","yarn.node-labels.fs-store.impl.class":"org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore","yarn.nodemanager.collector-service.address":"${yarn.nodemanager.hostname}:8048","fs.trash.checkpoint.interval":"0","mapreduce.job.map.output.collector.class":"org.apache.hadoop.mapred.MapTask$MapOutputBuffer","yarn.resourcemanager.node-ip-cache.expiry-interval-secs":"-1","hadoop.http.authentication.signature.secret.file":"*********(redacted)","hadoop.jetty.logs.serve.aliases":"true","yarn.resourcemanager.placement-constraints.handler":"disabled","yarn.timeline-service.handler-thread-count":"10","yarn.resourcemanager.max-completed-applications":"1000","yarn.resourcemanager.system-metrics-publisher.enabled":"false","yarn.resourcemanager.placement-constraints.algorithm.class":"org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.algorithm.DefaultPlacementAlgorithm","yarn.sharedcache.webapp.address":"0.0.0.0:8788","yarn.resourcemanager.delegation.token.renew-interval":"*********(redacted)","yarn.sharedcache.nm.uploader.replication.factor":"10","hadoop.security.groups.negative-cache.secs":"30","yarn.app.mapreduce.task.container.log.backups":"0","mapreduce.reduce.skip.proc-count.auto-incr":"true","hadoop.security.group.mapping.ldap.posix.attr.gid.name":"gidNumber","rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB":"org.apache.hadoop.ipc.ProtobufRpcEngine","ipc.client.fallback-to-simple-auth-allowed":"false","yarn.nodemanager.resource.memory.enforced":"true","hadoop.proxyuser.root.groups":"*","yarn.client.failover-proxy-provider":"org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider","yarn.timeline-service.http-authentication.simple.anonymous.allowed":"true","ha.health-monitor.check-interval.ms":"1000","yarn.acl.reservation-enable":"false","yarn.resourcemanager.store.class":"org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore","yarn.app.mapreduce.am.hard-kill-timeout-ms":"10000","fs.s3a.etag.checksum.enabled":"false","yarn.nodemanager.container-metrics.enable":"true","yarn.timeline-service.client.fd-clean-interval-secs":"60","yarn.resourcemanager.nodemanagers.heartbeat-interval-ms":"1000","hadoop.common.configuration.version":"3.0.0","fs.s3a.s3guard.ddb.table.capacity.read":"500","yarn.nodemanager.remote-app-log-dir-suffix":"logs","yarn.nodemanager.windows-container.cpu-limit.enabled":"false","yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed":"false","file.blocksize":"67108864","hadoop.registry.zk.retry.ceiling.ms":"60000","yarn.scheduler.configuration.leveldb-store.path":"${hadoop.tmp.dir}/yarn/system/confstore","yarn.sharedcache.store.in-memory.initial-delay-mins":"10","mapreduce.jobhistory.principal":"jhs/[email protected]","mapreduce.map.skip.proc-count.auto-incr":"true","fs.s3a.committer.name":"file","mapreduce.task.profile.reduces":"0-2","hadoop.zk.num-retries":"1000","yarn.webapp.xfs-filter.enabled":"true","seq.io.sort.mb":"100","yarn.scheduler.configuration.max.version":"100","yarn.timeline-service.webapp.https.address":"${yarn.timeline-service.hostname}:8190","yarn.resourcemanager.scheduler.address":"${yarn.resourcemanager.hostname}:8030","yarn.node-labels.enabled":"false","yarn.resourcemanager.webapp.ui-actions.enabled":"true","mapreduce.task.timeout":"600000","yarn.sharedcache.client-server.thread-count":"50","hadoop.security.groups.shell.command.timeout":"0s","hadoop.security.crypto.cipher.suite":"AES/CTR/NoPadding","yarn.nodemanager.elastic-memory-control.oom-handler":"org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.DefaultOOMHandler","yarn.resourcemanager.connect.max-wait.ms":"900000","fs.defaultFS":"hdfs://master:9000","yarn.minicluster.use-rpc":"false","fs.har.impl.disable.cache":"true","yarn.webapp.ui2.enable":"false","io.compression.codec.bzip2.library":"system-native","yarn.nodemanager.distributed-scheduling.enabled":"false","mapreduce.shuffle.connection-keep-alive.timeout":"5","yarn.resourcemanager.webapp.https.address":"${yarn.resourcemanager.hostname}:8090","mapreduce.jobhistory.address":"0.0.0.0:10020","yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs":"*********(redacted)","yarn.is.minicluster":"false","yarn.nodemanager.address":"${yarn.nodemanager.hostname}:0","fs.abfss.impl":"org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem","fs.AbstractFileSystem.s3a.impl":"org.apache.hadoop.fs.s3a.S3A","mapreduce.task.combine.progress.records":"10000","yarn.resourcemanager.epoch.range":"0","yarn.resourcemanager.am.max-attempts":"2","yarn.nodemanager.linux-container-executor.cgroups.hierarchy":"/hadoop-yarn","fs.AbstractFileSystem.wasbs.impl":"org.apache.hadoop.fs.azure.Wasbs","yarn.timeline-service.entity-group-fs-store.cache-store-class":"org.apache.hadoop.yarn.server.timeline.MemoryTimelineStore","fs.ftp.transfer.mode":"BLOCK_TRANSFER_MODE","ipc.server.log.slow.rpc":"false","yarn.resourcemanager.node-labels.provider.fetch-interval-ms":"1800000","yarn.router.webapp.https.address":"0.0.0.0:8091","yarn.nodemanager.webapp.cross-origin.enabled":"false","fs.wasb.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem","yarn.resourcemanager.auto-update.containers":"false","yarn.app.mapreduce.am.job.committer.cancel-timeout":"60000","yarn.scheduler.configuration.zk-store.parent-path":"/confstore","yarn.nodemanager.default-container-executor.log-dirs.permissions":"710","yarn.app.attempt.diagnostics.limit.kc":"64","ftp.bytes-per-checksum":"512","yarn.nodemanager.resource.memory-mb":"-1","fs.AbstractFileSystem.abfs.impl":"org.apache.hadoop.fs.azurebfs.Abfs","yarn.timeline-service.writer.flush-interval-seconds":"60","fs.s3a.fast.upload.active.blocks":"4","hadoop.security.credential.clear-text-fallback":"true","yarn.nodemanager.collector-service.thread-count":"5","fs.azure.secure.mode":"false","mapreduce.jobhistory.joblist.cache.size":"20000","fs.ftp.host":"0.0.0.0","yarn.resourcemanager.fs.state-store.num-retries":"0","yarn.resourcemanager.nodemanager-connect-retries":"10","yarn.nodemanager.log-aggregation.num-log-files-per-app":"30","hadoop.security.kms.client.encrypted.key.cache.low-watermark":"0.3f","fs.s3a.committer.magic.enabled":"false","yarn.timeline-service.client.max-retries":"30","dfs.ha.fencing.ssh.connect-timeout":"30000","yarn.log-aggregation-enable":"false","yarn.system-metrics-publisher.enabled":"false","mapreduce.reduce.markreset.buffer.percent":"0.0","fs.AbstractFileSystem.viewfs.impl":"org.apache.hadoop.fs.viewfs.ViewFs","mapreduce.task.io.sort.factor":"10","yarn.nodemanager.amrmproxy.client.thread-count":"25","ha.failover-controller.new-active.rpc-timeout.ms":"60000","yarn.nodemanager.container-localizer.java.opts":"-Xmx256m","mapreduce.jobhistory.datestring.cache.size":"200000","mapreduce.job.acl-modify-job":" ","yarn.nodemanager.windows-container.memory-limit.enabled":"false","yarn.timeline-service.webapp.address":"${yarn.timeline-service.hostname}:8188","yarn.app.mapreduce.am.job.committer.commit-window":"10000","yarn.nodemanager.container-manager.thread-count":"20","yarn.minicluster.fixed.ports":"false","hadoop.tags.system":"YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT\n      ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL","yarn.cluster.max-application-priority":"0","yarn.timeline-service.ttl-enable":"true","mapreduce.jobhistory.recovery.store.fs.uri":"${hadoop.tmp.dir}/mapred/history/recoverystore","hadoop.caller.context.signature.max.size":"40","yarn.client.load.resource-types.from-server":"false","ha.zookeeper.session-timeout.ms":"10000","tfile.io.chunk.size":"1048576","fs.s3a.s3guard.ddb.table.capacity.write":"100","mapreduce.job.speculative.slowtaskthreshold":"1.0","io.serializations":"org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization","hadoop.security.kms.client.failover.sleep.max.millis":"2000","hadoop.security.group.mapping.ldap.directory.search.timeout":"10000","yarn.scheduler.configuration.store.max-logs":"1000","yarn.nodemanager.node-attributes.provider.fetch-interval-ms":"600000","fs.swift.impl":"org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem","yarn.nodemanager.local-cache.max-files-per-directory":"8192","hadoop.http.cross-origin.enabled":"false","hadoop.zk.acl":"world:anyone:rwcda","mapreduce.map.sort.spill.percent":"0.80","yarn.timeline-service.entity-group-fs-store.scan-interval-seconds":"60","yarn.node-attribute.fs-store.impl.class":"org.apache.hadoop.yarn.server.resourcemanager.nodelabels.FileSystemNodeAttributeStore","fs.s3a.retry.interval":"500ms","yarn.timeline-service.client.best-effort":"false","yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled":"*********(redacted)","hadoop.security.group.mapping.ldap.posix.attr.uid.name":"uidNumber","fs.AbstractFileSystem.swebhdfs.impl":"org.apache.hadoop.fs.SWebHdfs","yarn.nodemanager.elastic-memory-control.timeout-sec":"5","mapreduce.ifile.readahead":"true","yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms":"300000","yarn.timeline-service.reader.webapp.address":"${yarn.timeline-service.webapp.address}","yarn.resourcemanager.placement-constraints.algorithm.pool-size":"1","yarn.timeline-service.hbase.coprocessor.jar.hdfs.location":"/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar","hadoop.security.kms.client.encrypted.key.cache.num.refill.threads":"2","yarn.resourcemanager.scheduler.class":"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler","yarn.app.mapreduce.am.command-opts":"-Xmx1024m","mapreduce.cluster.local.dir":"${hadoop.tmp.dir}/mapred/local","io.mapfile.bloom.error.rate":"0.005","fs.client.resolve.topology.enabled":"false","yarn.nodemanager.runtime.linux.allowed-runtimes":"default","yarn.sharedcache.store.class":"org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore","ha.failover-controller.graceful-fence.rpc-timeout.ms":"5000","ftp.replication":"3","hadoop.security.uid.cache.secs":"14400","mapreduce.job.maxtaskfailures.per.tracker":"3","fs.s3a.metadatastore.impl":"org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore","io.skip.checksum.errors":"false","yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts":"3","yarn.timeline-service.webapp.xfs-filter.xframe-options":"SAMEORIGIN","fs.s3a.connection.timeout":"200000","mapreduce.job.max.split.locations":"15","yarn.resourcemanager.nm-container-queuing.max-queue-length":"15","hadoop.registry.zk.session.timeout.ms":"60000","yarn.federation.cache-ttl.secs":"300","mapreduce.jvm.system-properties-to-log":"os.name,os.version,java.home,java.runtime.version,java.vendor,java.version,java.vm.name,java.class.path,java.io.tmpdir,user.dir,user.name","yarn.resourcemanager.opportunistic-container-allocation.nodes-used":"10","yarn.timeline-service.entity-group-fs-store.active-dir":"/tmp/entity-file-history/active","mapreduce.shuffle.transfer.buffer.size":"131072","yarn.timeline-service.client.retry-interval-ms":"1000","yarn.http.policy":"HTTP_ONLY","fs.s3a.socket.send.buffer":"8192","fs.AbstractFileSystem.abfss.impl":"org.apache.hadoop.fs.azurebfs.Abfss","yarn.sharedcache.uploader.server.address":"0.0.0.0:8046","yarn.resourcemanager.delegation-token.max-conf-size-bytes":"*********(redacted)","hadoop.http.authentication.token.validity":"*********(redacted)","mapreduce.shuffle.max.connections":"0","yarn.minicluster.yarn.nodemanager.resource.memory-mb":"4096","mapreduce.job.emit-timeline-data":"false","yarn.nodemanager.resource.system-reserved-memory-mb":"-1","hadoop.kerberos.min.seconds.before.relogin":"60","mapreduce.jobhistory.move.thread-count":"3","yarn.resourcemanager.admin.client.thread-count":"1","yarn.dispatcher.drain-events.timeout":"300000","fs.s3a.buffer.dir":"${hadoop.tmp.dir}/s3a","hadoop.ssl.enabled.protocols":"TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2","mapreduce.jobhistory.admin.address":"0.0.0.0:10033","yarn.log-aggregation-status.time-out.ms":"600000","fs.s3a.assumed.role.sts.endpoint.region":"us-west-1","mapreduce.shuffle.port":"13562","yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory":"10","yarn.nodemanager.health-checker.interval-ms":"600000","yarn.router.clientrm.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.router.clientrm.DefaultClientRequestInterceptor","yarn.resourcemanager.zk-appid-node.split-index":"0","ftp.blocksize":"67108864","yarn.nodemanager.runtime.linux.sandbox-mode.local-dirs.permissions":"read","yarn.router.rmadmin.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.router.rmadmin.DefaultRMAdminRequestInterceptor","yarn.nodemanager.log-container-debug-info.enabled":"true","yarn.client.max-cached-nodemanagers-proxies":"0","yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms":"20","yarn.nodemanager.delete.debug-delay-sec":"0","yarn.nodemanager.pmem-check-enabled":"true","yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage":"90.0","mapreduce.app-submission.cross-platform":"false","yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms":"10000","yarn.nodemanager.container-retry-minimum-interval-ms":"1000","hadoop.security.groups.cache.secs":"300","yarn.federation.enabled":"false","fs.azure.local.sas.key.mode":"false","ipc.maximum.data.length":"67108864","mapreduce.shuffle.max.threads":"0","yarn.router.pipeline.cache-max-size":"25","yarn.resourcemanager.nm-container-queuing.load-comparator":"QUEUE_LENGTH","hadoop.security.authorization":"false","mapreduce.job.complete.cancel.delegation.tokens":"*********(redacted)","fs.s3a.paging.maximum":"5000","nfs.exports.allowed.hosts":"* rw","yarn.nodemanager.amrmproxy.ha.enable":"false","mapreduce.jobhistory.http.policy":"HTTP_ONLY","yarn.sharedcache.store.in-memory.check-period-mins":"720","hadoop.security.group.mapping.ldap.ssl":"false","yarn.client.application-client-protocol.poll-interval-ms":"200","yarn.scheduler.configuration.leveldb-store.compaction-interval-secs":"86400","yarn.timeline-service.writer.class":"org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl","ha.zookeeper.parent-znode":"/hadoop-ha","yarn.nodemanager.log-aggregation.policy.class":"org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy","mapreduce.reduce.shuffle.merge.percent":"0.66","hadoop.security.group.mapping.ldap.search.filter.group":"(objectClass=group)","yarn.resourcemanager.placement-constraints.scheduler.pool-size":"1","yarn.nodemanager.resourcemanager.minimum.version":"NONE","mapreduce.job.speculative.speculative-cap-running-tasks":"0.1","yarn.admin.acl":"*","yarn.nodemanager.recovery.supervised":"false","yarn.sharedcache.admin.thread-count":"1","yarn.resourcemanager.ha.automatic-failover.enabled":"true","mapreduce.reduce.skip.maxgroups":"0","mapreduce.reduce.shuffle.connect.timeout":"180000","yarn.resourcemanager.address":"${yarn.resourcemanager.hostname}:8032","ipc.client.ping":"true","mapreduce.task.local-fs.write-limit.bytes":"-1","fs.adl.oauth2.access.token.provider.type":"*********(redacted)","mapreduce.shuffle.ssl.file.buffer.size":"65536","yarn.resourcemanager.ha.automatic-failover.embedded":"true","yarn.nodemanager.resource-plugins.gpu.docker-plugin":"nvidia-docker-v1","hadoop.ssl.enabled":"false","fs.s3a.multipart.purge":"false","yarn.scheduler.configuration.store.class":"file","yarn.resourcemanager.nm-container-queuing.queue-limit-stdev":"1.0f","mapreduce.job.end-notification.max.attempts":"5","mapreduce.output.fileoutputformat.compress.codec":"org.apache.hadoop.io.compress.DefaultCodec","yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled":"false","ipc.client.bind.wildcard.addr":"false","yarn.resourcemanager.webapp.rest-csrf.enabled":"false","ha.health-monitor.connect-retry-interval.ms":"1000","yarn.nodemanager.keytab":"/etc/krb5.keytab","mapreduce.jobhistory.keytab":"/etc/security/keytab/jhs.service.keytab","fs.s3a.threads.max":"10","mapreduce.reduce.shuffle.input.buffer.percent":"0.70","yarn.nodemanager.runtime.linux.docker.allowed-container-networks":"host,none,bridge","yarn.nodemanager.node-labels.resync-interval-ms":"120000","hadoop.tmp.dir":"/home/hadoop-3.2.1/tmp/","mapreduce.job.maps":"2","mapreduce.jobhistory.webapp.rest-csrf.custom-header":"X-XSRF-Header","mapreduce.job.end-notification.max.retry.interval":"5000","yarn.log-aggregation.retain-check-interval-seconds":"-1","yarn.resourcemanager.resource-tracker.client.thread-count":"50","yarn.rm.system-metrics-publisher.emit-container-events":"false","yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size":"10000","yarn.resourcemanager.ha.automatic-failover.zk-base-path":"/yarn-leader-election","io.seqfile.local.dir":"${hadoop.tmp.dir}/io/local","fs.s3a.s3guard.ddb.throttle.retry.interval":"100ms","fs.AbstractFileSystem.wasb.impl":"org.apache.hadoop.fs.azure.Wasb","mapreduce.client.submit.file.replication":"10","mapreduce.jobhistory.minicluster.fixed.ports":"false","fs.s3a.multipart.threshold":"2147483647","yarn.resourcemanager.webapp.xfs-filter.xframe-options":"SAMEORIGIN","mapreduce.jobhistory.done-dir":"${yarn.app.mapreduce.am.staging-dir}/history/done","dfs.namenode.name.dir":"/data/hadoop/hdfs/name","ipc.client.idlethreshold":"4000","yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage":"false","mapreduce.reduce.input.buffer.percent":"0.0","yarn.nodemanager.runtime.linux.docker.userremapping-gid-threshold":"1","yarn.nodemanager.webapp.rest-csrf.enabled":"false","fs.ftp.host.port":"21","ipc.ping.interval":"60000","yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size":"10","yarn.resourcemanager.admin.address":"${yarn.resourcemanager.hostname}:8033","file.client-write-packet-size":"65536","ipc.client.kill.max":"10","mapreduce.reduce.speculative":"true","hadoop.security.key.default.bitlength":"128","mapreduce.job.reducer.unconditional-preempt.delay.sec":"300","yarn.nodemanager.disk-health-checker.interval-ms":"120000","yarn.nodemanager.log.deletion-threads-count":"4","yarn.webapp.filter-entity-list-by-user":"false","ipc.client.connection.maxidletime":"10000","mapreduce.task.io.sort.mb":"100","yarn.nodemanager.localizer.client.thread-count":"5","io.erasurecode.codec.rs.rawcoders":"rs_native,rs_java","io.erasurecode.codec.rs-legacy.rawcoders":"rs-legacy_java","yarn.sharedcache.admin.address":"0.0.0.0:8047","yarn.resourcemanager.placement-constraints.algorithm.iterator":"SERIAL","yarn.nodemanager.localizer.cache.cleanup.interval-ms":"600000","hadoop.security.crypto.codec.classes.aes.ctr.nopadding":"org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec","mapreduce.job.cache.limit.max-resources-mb":"0","fs.s3a.connection.ssl.enabled":"true","yarn.nodemanager.process-kill-wait.ms":"5000","mapreduce.job.hdfs-servers":"${fs.defaultFS}","hadoop.workaround.non.threadsafe.getpwuid":"true","fs.df.interval":"60000","fs.s3a.multiobjectdelete.enable":"true","yarn.sharedcache.cleaner.resource-sleep-ms":"0","yarn.nodemanager.disk-health-checker.min-healthy-disks":"0.25","hadoop.shell.missing.defaultFs.warning":"false","io.file.buffer.size":"65536","hadoop.security.group.mapping.ldap.search.attr.member":"member","hadoop.security.random.device.file.path":"/dev/urandom","hadoop.security.sensitive-config-keys":"*********(redacted)","fs.s3a.s3guard.ddb.max.retries":"9","hadoop.rpc.socket.factory.class.default":"org.apache.hadoop.net.StandardSocketFactory","dfs.permissions.enabled":"false","yarn.intermediate-data-encryption.enable":"false","yarn.resourcemanager.connect.retry-interval.ms":"30000","yarn.nodemanager.container.stderr.pattern":"{*stderr*,*STDERR*}","yarn.scheduler.minimum-allocation-mb":"1024","yarn.app.mapreduce.am.staging-dir":"/tmp/hadoop-yarn/staging","mapreduce.reduce.shuffle.read.timeout":"180000","hadoop.http.cross-origin.max-age":"1800","io.erasurecode.codec.xor.rawcoders":"xor_native,xor_java","fs.s3a.connection.establish.timeout":"5000","mapreduce.job.running.map.limit":"0","yarn.minicluster.control-resource-monitoring":"false","hadoop.ssl.require.client.cert":"false","hadoop.kerberos.kinit.command":"kinit","yarn.federation.state-store.class":"org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore","mapreduce.reduce.log.level":"INFO","hadoop.security.dns.log-slow-lookups.threshold.ms":"1000","mapreduce.job.ubertask.enable":"false","adl.http.timeout":"-1","yarn.resourcemanager.placement-constraints.retry-attempts":"3","hadoop.caller.context.enabled":"false","yarn.nodemanager.vmem-pmem-ratio":"2.1","hadoop.rpc.protection":"authentication","ha.health-monitor.rpc-timeout.ms":"45000","yarn.nodemanager.remote-app-log-dir":"/tmp/logs","hadoop.zk.timeout-ms":"10000","fs.s3a.s3guard.cli.prune.age":"86400000","yarn.nodemanager.resource.pcores-vcores-multiplier":"1.0","yarn.nodemanager.runtime.linux.sandbox-mode":"disabled","yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size":"10","fs.s3a.committer.threads":"8","hadoop.zk.retry-interval-ms":"1000","hadoop.security.crypto.buffer.size":"8192","yarn.nodemanager.node-labels.provider.fetch-interval-ms":"600000","mapreduce.jobhistory.recovery.store.leveldb.path":"${hadoop.tmp.dir}/mapred/history/recoverystore","yarn.client.failover-retries-on-socket-timeouts":"0","yarn.nodemanager.resource.memory.enabled":"false","fs.azure.authorization.caching.enable":"true","hadoop.security.instrumentation.requires.admin":"false","yarn.nodemanager.delete.thread-count":"4","mapreduce.job.finish-when-all-reducers-done":"true","hadoop.registry.jaas.context":"Client","yarn.timeline-service.leveldb-timeline-store.path":"${hadoop.tmp.dir}/yarn/timeline","io.map.index.interval":"128","yarn.resourcemanager.nm-container-queuing.max-queue-wait-time-ms":"100","fs.abfs.impl":"org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem","mapreduce.job.counters.max":"120","mapreduce.jobhistory.webapp.rest-csrf.enabled":"false","yarn.timeline-service.store-class":"org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore","mapreduce.jobhistory.move.interval-ms":"180000","yarn.nodemanager.localizer.fetch.thread-count":"4","yarn.resourcemanager.scheduler.client.thread-count":"50","hadoop.ssl.hostname.verifier":"DEFAULT","yarn.timeline-service.leveldb-state-store.path":"${hadoop.tmp.dir}/yarn/timeline","mapreduce.job.classloader":"false","mapreduce.task.profile.map.params":"${mapreduce.task.profile.params}","ipc.client.connect.timeout":"20000","hadoop.security.auth_to_local.mechanism":"hadoop","yarn.timeline-service.app-collector.linger-period.ms":"60000","yarn.nm.liveness-monitor.expiry-interval-ms":"600000","yarn.resourcemanager.reservation-system.planfollower.time-step":"1000","yarn.nodemanager.runtime.linux.docker.enable-userremapping.allowed":"true","yarn.webapp.api-service.enable":"false","yarn.nodemanager.recovery.enabled":"false","mapreduce.job.end-notification.retry.interval":"1000","fs.du.interval":"600000","fs.ftp.impl":"org.apache.hadoop.fs.ftp.FTPFileSystem","yarn.nodemanager.container.stderr.tail.bytes":"4096","hadoop.security.group.mapping.ldap.read.timeout.ms":"60000","hadoop.security.groups.cache.warn.after.ms":"5000","file.bytes-per-checksum":"512","mapreduce.outputcommitter.factory.scheme.s3a":"org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory","hadoop.security.groups.cache.background.reload":"false","yarn.nodemanager.container-monitor.enabled":"true","yarn.nodemanager.elastic-memory-control.enabled":"false","net.topology.script.number.args":"100","mapreduce.task.merge.progress.records":"10000","yarn.nodemanager.localizer.address":"${yarn.nodemanager.hostname}:8040","yarn.timeline-service.keytab":"/etc/krb5.keytab","mapreduce.reduce.shuffle.fetch.retry.timeout-ms":"30000","yarn.resourcemanager.rm.container-allocation.expiry-interval-ms":"600000","mapreduce.fileoutputcommitter.algorithm.version":"2","yarn.resourcemanager.work-preserving-recovery.enabled":"true","mapreduce.map.skip.maxrecords":"0","yarn.sharedcache.root-dir":"/sharedcache","fs.s3a.retry.throttle.limit":"${fs.s3a.attempts.maximum}","hadoop.http.authentication.type":"simple","mapreduce.job.cache.limit.max-resources":"0","mapreduce.task.userlog.limit.kb":"0","yarn.resourcemanager.scheduler.monitor.enable":"false","ipc.client.connect.max.retries":"10","hadoop.registry.zk.retry.times":"5","dfs.namenode.http-address":"master:50070","yarn.nodemanager.resource-monitor.interval-ms":"3000","yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices":"auto","mapreduce.job.sharedcache.mode":"disabled","yarn.nodemanager.webapp.rest-csrf.custom-header":"X-XSRF-Header","mapreduce.shuffle.listen.queue.size":"128","yarn.scheduler.configuration.mutation.acl-policy.class":"org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy","mapreduce.map.cpu.vcores":"1","yarn.log-aggregation.file-formats":"TFile","yarn.timeline-service.client.fd-retain-secs":"300","hadoop.user.group.static.mapping.overrides":"dr.who=;","fs.azure.sas.expiry.period":"90d","mapreduce.jobhistory.recovery.store.class":"org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService","yarn.resourcemanager.fail-fast":"${yarn.fail-fast}","yarn.resourcemanager.proxy-user-privileges.enabled":"false","yarn.router.webapp.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.router.webapp.DefaultRequestInterceptorREST","yarn.nodemanager.resource.memory.cgroups.soft-limit-percentage":"90.0","mapreduce.job.reducer.preempt.delay.sec":"0","hadoop.util.hash.type":"murmur","yarn.nodemanager.disk-validator":"basic","yarn.app.mapreduce.client.job.max-retries":"3","mapreduce.reduce.shuffle.retry-delay.max.ms":"60000","hadoop.security.group.mapping.ldap.connection.timeout.ms":"60000","mapreduce.task.profile.params":"-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s","yarn.app.mapreduce.shuffle.log.backups":"0","yarn.nodemanager.container-diagnostics-maximum-size":"10000","hadoop.registry.zk.retry.interval.ms":"1000","yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms":"1000","fs.AbstractFileSystem.file.impl":"org.apache.hadoop.fs.local.LocalFs","yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds":"-1","mapreduce.jobhistory.cleaner.interval-ms":"86400000","hadoop.registry.zk.quorum":"localhost:2181","mapreduce.output.fileoutputformat.compress":"false","yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs":"*********(redacted)","fs.s3a.assumed.role.session.duration":"30m","hadoop.security.group.mapping.ldap.conversion.rule":"none","hadoop.ssl.server.conf":"ssl-server.xml","fs.s3a.retry.throttle.interval":"1000ms","seq.io.sort.factor":"100","yarn.sharedcache.cleaner.initial-delay-mins":"10","mapreduce.client.completion.pollinterval":"5000","hadoop.ssl.keystores.factory.class":"org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory","yarn.app.mapreduce.am.resource.cpu-vcores":"1","yarn.timeline-service.enabled":"false","yarn.nodemanager.runtime.linux.docker.capabilities":"CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE","yarn.acl.enable":"false","yarn.timeline-service.entity-group-fs-store.done-dir":"/tmp/entity-file-history/done/","mapreduce.task.profile":"false","yarn.resourcemanager.fs.state-store.uri":"${hadoop.tmp.dir}/yarn/system/rmstore","mapreduce.jobhistory.always-scan-user-dir":"false","yarn.nodemanager.opportunistic-containers-use-pause-for-preemption":"false","yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user":"nobody","yarn.timeline-service.reader.class":"org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl","yarn.resourcemanager.configuration.provider-class":"org.apache.hadoop.yarn.LocalConfigurationProvider","yarn.nodemanager.runtime.linux.docker.userremapping-uid-threshold":"1","yarn.resourcemanager.configuration.file-system-based-store":"/yarn/conf","mapreduce.job.cache.limit.max-single-resource-mb":"0","yarn.nodemanager.runtime.linux.docker.stop.grace-period":"10","yarn.resourcemanager.resource-profiles.source-file":"resource-profiles.json","yarn.nodemanager.resource.percentage-physical-cpu-limit":"100","mapreduce.jobhistory.client.thread-count":"10","tfile.fs.input.buffer.size":"262144","mapreduce.client.progressmonitor.pollinterval":"1000","yarn.nodemanager.log-dirs":"${yarn.log.dir}/userlogs","fs.automatic.close":"true","yarn.nodemanager.hostname":"0.0.0.0","yarn.nodemanager.resource.memory.cgroups.swappiness":"0","ftp.stream-buffer-size":"4096","yarn.fail-fast":"false","yarn.timeline-service.app-aggregation-interval-secs":"15","hadoop.security.group.mapping.ldap.search.filter.user":"(&(objectClass=user)(sAMAccountName={0}))","yarn.nodemanager.container-localizer.log.level":"INFO","yarn.timeline-service.address":"${yarn.timeline-service.hostname}:10200","mapreduce.job.ubertask.maxmaps":"9","fs.s3a.threads.keepalivetime":"60","mapreduce.jobhistory.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","mapreduce.task.files.preserve.failedtasks":"false","yarn.app.mapreduce.client.job.retry-interval":"2000","ha.failover-controller.graceful-fence.connection.retries":"1","yarn.resourcemanager.delegation.token.max-lifetime":"*********(redacted)","yarn.timeline-service.client.drain-entities.timeout.ms":"2000","yarn.nodemanager.resource-plugins.fpga.vendor-plugin.class":"org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.IntelFpgaOpenclPlugin","yarn.timeline-service.entity-group-fs-store.summary-store":"org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore","mapreduce.reduce.cpu.vcores":"1","mapreduce.job.encrypted-intermediate-data.buffer.kb":"128","fs.client.resolve.remote.symlinks":"true","yarn.nodemanager.webapp.https.address":"0.0.0.0:8044","hadoop.http.cross-origin.allowed-origins":"*","mapreduce.job.encrypted-intermediate-data":"false","yarn.timeline-service.entity-group-fs-store.retain-seconds":"604800","yarn.resourcemanager.metrics.runtime.buckets":"60,300,1440","yarn.timeline-service.generic-application-history.max-applications":"10000","yarn.nodemanager.local-dirs":"${hadoop.tmp.dir}/nm-local-dir","mapreduce.shuffle.connection-keep-alive.enable":"false","yarn.node-labels.configuration-type":"centralized","fs.s3a.path.style.access":"false","yarn.nodemanager.aux-services.mapreduce_shuffle.class":"org.apache.hadoop.mapred.ShuffleHandler","yarn.sharedcache.store.in-memory.staleness-period-mins":"10080","fs.adl.impl":"org.apache.hadoop.fs.adl.AdlFileSystem","yarn.resourcemanager.nodemanager.minimum.version":"NONE","mapreduce.jobhistory.webapp.xfs-filter.xframe-options":"SAMEORIGIN","yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled":"false","net.topology.impl":"org.apache.hadoop.net.NetworkTopology","io.map.index.skip":"0","yarn.timeline-service.reader.webapp.https.address":"${yarn.timeline-service.webapp.https.address}","fs.ftp.data.connection.mode":"ACTIVE_LOCAL_DATA_CONNECTION_MODE","mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed":"true","yarn.scheduler.maximum-allocation-vcores":"4","hadoop.http.cross-origin.allowed-headers":"X-Requested-With,Content-Type,Accept,Origin","yarn.nodemanager.log-aggregation.compression-type":"none","yarn.timeline-service.version":"1.0f","yarn.ipc.rpc.class":"org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC","mapreduce.reduce.maxattempts":"4","hadoop.security.dns.log-slow-lookups.enabled":"false","mapreduce.job.committer.setup.cleanup.needed":"true","mapreduce.job.running.reduce.limit":"0","ipc.maximum.response.length":"134217728","yarn.resourcemanager.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","mapreduce.job.token.tracking.ids.enabled":"*********(redacted)","hadoop.caller.context.max.size":"128","yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed":"false","yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed":"false","hadoop.registry.system.acls":"sasl:yarn@, sasl:mapred@, sasl:hdfs@","yarn.nodemanager.recovery.dir":"${hadoop.tmp.dir}/yarn-nm-recovery","fs.s3a.fast.upload.buffer":"disk","mapreduce.jobhistory.intermediate-done-dir":"${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate","yarn.app.mapreduce.shuffle.log.separate":"true","fs.s3a.max.total.tasks":"5","fs.s3a.readahead.range":"64K","hadoop.http.authentication.simple.anonymous.allowed":"true","fs.s3a.attempts.maximum":"20","hadoop.registry.zk.connection.timeout.ms":"15000","yarn.resourcemanager.delegation-token-renewer.thread-count":"*********(redacted)","yarn.nodemanager.health-checker.script.timeout-ms":"1200000","yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size":"10000","yarn.resourcemanager.resource-profiles.enabled":"false","yarn.timeline-service.hbase-schema.prefix":"prod.","fs.azure.authorization":"false","mapreduce.map.log.level":"INFO","yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs":"20","mapreduce.output.fileoutputformat.compress.type":"RECORD","yarn.resourcemanager.leveldb-state-store.path":"${hadoop.tmp.dir}/yarn/system/rmstore","yarn.timeline-service.webapp.rest-csrf.custom-header":"X-XSRF-Header","mapreduce.ifile.readahead.bytes":"4194304","yarn.sharedcache.app-checker.class":"org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker","yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users":"true","yarn.nodemanager.resource.detect-hardware-capabilities":"false","mapreduce.cluster.acls.enabled":"false","mapreduce.job.speculative.retry-after-no-speculate":"1000","hadoop.security.group.mapping.ldap.search.group.hierarchy.levels":"0","yarn.resourcemanager.fs.state-store.retry-interval-ms":"1000","file.stream-buffer-size":"4096","yarn.resourcemanager.application-timeouts.monitor.interval-ms":"3000","mapreduce.map.output.compress.codec":"org.apache.hadoop.io.compress.DefaultCodec","mapreduce.map.speculative":"true","mapreduce.job.speculative.retry-after-speculate":"15000","yarn.nodemanager.linux-container-executor.cgroups.mount":"false","yarn.app.mapreduce.am.container.log.backups":"0","yarn.app.mapreduce.am.log.level":"INFO","mapreduce.job.reduce.slowstart.completedmaps":"0.05","yarn.timeline-service.http-authentication.type":"simple","hadoop.security.group.mapping.ldap.search.attr.group.name":"cn","yarn.nodemanager.resource-plugins.fpga.allowed-fpga-devices":"0,1","yarn.timeline-service.client.internal-timers-ttl-secs":"420","hadoop.http.logs.enabled":"true","fs.s3a.block.size":"32M","yarn.sharedcache.client-server.address":"0.0.0.0:8045","yarn.nodemanager.logaggregation.threadpool-size-max":"100","yarn.resourcemanager.hostname":"master","yarn.resourcemanager.delegation.key.update-interval":"86400000","mapreduce.reduce.shuffle.fetch.retry.enabled":"${yarn.nodemanager.recovery.enabled}","mapreduce.map.memory.mb":"-1","mapreduce.task.skip.start.attempts":"2","fs.AbstractFileSystem.hdfs.impl":"org.apache.hadoop.fs.Hdfs","yarn.nodemanager.disk-health-checker.enable":"true","ipc.client.tcpnodelay":"true","ipc.client.rpc-timeout.ms":"0","yarn.nodemanager.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","ipc.client.low-latency":"false","mapreduce.input.lineinputformat.linespermap":"1","yarn.router.interceptor.user.threadpool-size":"5","ipc.client.connect.max.retries.on.timeouts":"45","yarn.timeline-service.leveldb-timeline-store.read-cache-size":"104857600","fs.AbstractFileSystem.har.impl":"org.apache.hadoop.fs.HarFs","mapreduce.job.split.metainfo.maxsize":"10000000","yarn.am.liveness-monitor.expiry-interval-ms":"600000","yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs":"*********(redacted)","yarn.timeline-service.entity-group-fs-store.app-cache-size":"10","fs.s3a.socket.recv.buffer":"8192","yarn.resourcemanager.resource-tracker.address":"${yarn.resourcemanager.hostname}:8031","yarn.nodemanager.node-labels.provider.fetch-timeout-ms":"1200000","mapreduce.job.heap.memory-mb.ratio":"0.8","yarn.resourcemanager.leveldb-state-store.compaction-interval-secs":"3600","yarn.resourcemanager.webapp.rest-csrf.custom-header":"X-XSRF-Header","yarn.scheduler.configuration.fs.path":"file://${hadoop.tmp.dir}/yarn/system/schedconf","mapreduce.client.output.filter":"FAILED","hadoop.http.filter.initializers":"org.apache.hadoop.http.lib.StaticUserWebFilter","mapreduce.reduce.memory.mb":"-1","yarn.timeline-service.hostname":"0.0.0.0","file.replication":"1","yarn.nodemanager.container-metrics.unregister-delay-ms":"10000","yarn.nodemanager.container-metrics.period-ms":"-1","mapreduce.fileoutputcommitter.task.cleanup.enabled":"false","yarn.nodemanager.log.retain-seconds":"10800","yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds":"3600","yarn.resourcemanager.keytab":"/etc/krb5.keytab","hadoop.security.group.mapping.providers.combined":"true","mapreduce.reduce.merge.inmem.threshold":"1000","yarn.timeline-service.recovery.enabled":"false","fs.azure.saskey.usecontainersaskeyforallaccess":"true","yarn.sharedcache.nm.uploader.thread-count":"20","yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs":"3600","mapreduce.shuffle.ssl.enabled":"false","yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds":"259200000","fs.s3a.committer.staging.abort.pending.uploads":"true","yarn.nodemanager.opportunistic-containers-max-queue-length":"0","yarn.resourcemanager.state-store.max-completed-applications":"${yarn.resourcemanager.max-completed-applications}","mapreduce.job.speculative.minimum-allowed-tasks":"10","yarn.log-aggregation.retain-seconds":"-1","yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb":"0","mapreduce.jobhistory.max-age-ms":"604800000","hadoop.http.cross-origin.allowed-methods":"GET,POST,HEAD","yarn.resourcemanager.opportunistic-container-allocation.enabled":"false","mapreduce.jobhistory.webapp.address":"0.0.0.0:19888","hadoop.system.tags":"YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT\n      ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL","yarn.log-aggregation.file-controller.TFile.class":"org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController","yarn.client.nodemanager-connect.max-wait-ms":"180000","yarn.resourcemanager.webapp.address":"${yarn.resourcemanager.hostname}:8088","mapreduce.jobhistory.recovery.enable":"false","mapreduce.reduce.shuffle.parallelcopies":"5","fs.AbstractFileSystem.webhdfs.impl":"org.apache.hadoop.fs.WebHdfs","fs.trash.interval":"0","yarn.app.mapreduce.client.max-retries":"3","hadoop.security.authentication":"simple","mapreduce.task.profile.reduce.params":"${mapreduce.task.profile.params}","yarn.app.mapreduce.am.resource.mb":"1536","mapreduce.input.fileinputformat.list-status.num-threads":"1","yarn.nodemanager.container-executor.class":"org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor","io.mapfile.bloom.size":"1048576","yarn.timeline-service.ttl-ms":"604800000","yarn.resourcemanager.nm-container-queuing.min-queue-length":"5","yarn.nodemanager.resource.cpu-vcores":"-1","mapreduce.job.reduces":"1","fs.s3a.multipart.size":"100M","yarn.scheduler.minimum-allocation-vcores":"1","mapreduce.job.speculative.speculative-cap-total-tasks":"0.01","hadoop.ssl.client.conf":"ssl-client.xml","mapreduce.job.queuename":"default","mapreduce.job.encrypted-intermediate-data-key-size-bits":"128","fs.s3a.metadatastore.authoritative":"false","yarn.nodemanager.webapp.xfs-filter.xframe-options":"SAMEORIGIN","ha.health-monitor.sleep-after-disconnect.ms":"1000","yarn.app.mapreduce.shuffle.log.limit.kb":"0","hadoop.security.group.mapping":"org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback","yarn.client.application-client-protocol.poll-timeout-ms":"-1","mapreduce.jobhistory.jhist.format":"binary","yarn.resourcemanager.ha.enabled":"false","hadoop.http.staticuser.user":"dr.who","mapreduce.task.exit.timeout.check-interval-ms":"20000","mapreduce.jobhistory.intermediate-user-done-dir.permissions":"770","mapreduce.task.exit.timeout":"60000","yarn.nodemanager.linux-container-executor.resources-handler.class":"org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler","mapreduce.reduce.shuffle.memory.limit.percent":"0.25","yarn.resourcemanager.reservation-system.enable":"false","mapreduce.map.output.compress":"false","ha.zookeeper.acl":"world:anyone:rwcda","ipc.server.max.connections":"0","yarn.nodemanager.aux-services":"mapreduce_shuffle","yarn.nodemanager.runtime.linux.docker.default-container-network":"host","yarn.router.webapp.address":"0.0.0.0:8089","yarn.scheduler.maximum-allocation-mb":"8192","yarn.resourcemanager.scheduler.monitor.policies":"org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy","yarn.sharedcache.cleaner.period-mins":"1440","yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint":"http://localhost:3476/v1.0/docker/cli","yarn.app.mapreduce.am.container.log.limit.kb":"0","ipc.client.connect.retry.interval":"1000","yarn.timeline-service.http-cross-origin.enabled":"false","fs.wasbs.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure","yarn.federation.subcluster-resolver.class":"org.apache.hadoop.yarn.server.federation.resolver.DefaultSubClusterResolverImpl","yarn.resourcemanager.zk-state-store.parent-path":"/rmstore","mapreduce.jobhistory.cleaner.enable":"true","yarn.timeline-service.client.fd-flush-interval-secs":"10","hadoop.security.kms.client.encrypted.key.cache.expiry":"43200000","yarn.client.nodemanager-client-async.thread-pool-max-size":"500","mapreduce.map.maxattempts":"4","yarn.resourcemanager.nm-container-queuing.sorting-nodes-interval-ms":"1000","fs.s3a.committer.staging.tmp.path":"tmp/staging","yarn.nodemanager.sleep-delay-before-sigkill.ms":"250","yarn.resourcemanager.nm-container-queuing.min-queue-wait-time-ms":"10","mapreduce.job.end-notification.retry.attempts":"0","yarn.nodemanager.resource.count-logical-processors-as-cores":"false","hadoop.registry.zk.root":"/registry","adl.feature.ownerandgroup.enableupn":"false","yarn.resourcemanager.zk-max-znode-size.bytes":"1048576","mapreduce.job.reduce.shuffle.consumer.plugin.class":"org.apache.hadoop.mapreduce.task.reduce.Shuffle","yarn.resourcemanager.delayed.delegation-token.removal-interval-ms":"*********(redacted)","yarn.nodemanager.localizer.cache.target-size-mb":"10240","fs.s3a.committer.staging.conflict-mode":"fail","mapreduce.client.libjars.wildcard":"true","fs.s3a.committer.staging.unique-filenames":"true","yarn.nodemanager.node-attributes.provider.fetch-timeout-ms":"1200000","fs.s3a.list.version":"2","ftp.client-write-packet-size":"65536","fs.AbstractFileSystem.adl.impl":"org.apache.hadoop.fs.adl.Adl","hadoop.security.key.default.cipher":"AES/CTR/NoPadding","yarn.client.failover-retries":"0","fs.s3a.multipart.purge.age":"86400","mapreduce.job.local-fs.single-disk-limit.check.interval-ms":"5000","net.topology.node.switch.mapping.impl":"org.apache.hadoop.net.ScriptBasedMapping","yarn.nodemanager.amrmproxy.address":"0.0.0.0:8049","ipc.server.listen.queue.size":"128","map.sort.class":"org.apache.hadoop.util.QuickSort","fs.viewfs.rename.strategy":"SAME_MOUNTPOINT","hadoop.security.kms.client.authentication.retry-count":"1","fs.permissions.umask-mode":"022","fs.s3a.assumed.role.credentials.provider":"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider","yarn.nodemanager.vmem-check-enabled":"true","yarn.nodemanager.numa-awareness.enabled":"false","yarn.nodemanager.recovery.compaction-interval-secs":"3600","yarn.app.mapreduce.client-am.ipc.max-retries":"3","yarn.federation.registry.base-dir":"yarnfederation/","mapreduce.job.max.map":"-1","mapreduce.job.local-fs.single-disk-limit.bytes":"-1","mapreduce.job.ubertask.maxreduces":"1","hadoop.security.kms.client.encrypted.key.cache.size":"500","hadoop.security.java.secure.random.algorithm":"SHA1PRNG","ha.failover-controller.cli-check.rpc-timeout.ms":"20000","mapreduce.jobhistory.jobname.limit":"50","mapreduce.application.classpath":"$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*","yarn.client.nodemanager-connect.retry-interval-ms":"10000","yarn.timeline-service.state-store-class":"org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore","yarn.nodemanager.env-whitelist":"JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME","yarn.sharedcache.nested-level":"3","yarn.timeline-service.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","fs.azure.user.agent.prefix":"unknown","yarn.resourcemanager.zk-delegation-token-node.split-index":"*********(redacted)","yarn.nodemanager.numa-awareness.read-topology":"false","yarn.nodemanager.webapp.address":"${yarn.nodemanager.hostname}:8042","rpc.metrics.quantile.enable":"false","yarn.registry.class":"org.apache.hadoop.registry.client.impl.FSRegistryOperationsService","mapreduce.jobhistory.admin.acl":"*","yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size":"10","yarn.scheduler.queue-placement-rules":"user-group","hadoop.http.authentication.kerberos.keytab":"${user.home}/hadoop.keytab","yarn.resourcemanager.recovery.enabled":"false","yarn.timeline-service.webapp.rest-csrf.enabled":"false"},"System Properties":{"java.io.tmpdir":"/tmp","line.separator":"\n","path.separator":":","sun.management.compiler":"HotSpot 64-Bit Tiered Compilers","SPARK_SUBMIT":"true","sun.cpu.endian":"little","java.specification.version":"1.8","java.vm.specification.name":"Java Virtual Machine Specification","java.vendor":"Oracle Corporation","java.vm.specification.version":"1.8","user.home":"/root","file.encoding.pkg":"sun.io","sun.nio.ch.bugLevel":"","sun.arch.data.model":"64","sun.boot.library.path":"/home/java/jre/lib/amd64","user.dir":"/home/spark-3.0.1","java.library.path":"/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib","sun.cpu.isalist":"","os.arch":"amd64","java.vm.version":"25.162-b12","jetty.git.hash":"e1bc35120a6617ee3df052294e433f3a25ce7097","java.endorsed.dirs":"/home/java/jre/lib/endorsed","java.runtime.version":"1.8.0_162-b12","java.vm.info":"mixed mode","java.ext.dirs":"/home/java/jre/lib/ext:/usr/java/packages/lib/ext","java.runtime.name":"Java(TM) SE Runtime Environment","file.separator":"/","java.class.version":"52.0","java.specification.name":"Java Platform API Specification","sun.boot.class.path":"/home/java/jre/lib/resources.jar:/home/java/jre/lib/rt.jar:/home/java/jre/lib/sunrsasign.jar:/home/java/jre/lib/jsse.jar:/home/java/jre/lib/jce.jar:/home/java/jre/lib/charsets.jar:/home/java/jre/lib/jfr.jar:/home/java/jre/classes","file.encoding":"UTF-8","user.timezone":"Asia/Shanghai","java.specification.vendor":"Oracle Corporation","sun.java.launcher":"SUN_STANDARD","os.version":"3.10.0-1062.el7.x86_64","sun.os.patch.level":"unknown","java.vm.specification.vendor":"Oracle Corporation","user.country":"CN","sun.jnu.encoding":"UTF-8","user.language":"zh","java.vendor.url":"http://java.oracle.com/","java.awt.printerjob":"sun.print.PSPrinterJob","java.awt.graphicsenv":"sun.awt.X11GraphicsEnvironment","awt.toolkit":"sun.awt.X11.XToolkit","os.name":"Linux","java.vm.vendor":"Oracle Corporation","java.vendor.url.bug":"http://bugreport.sun.com/bugreport/","user.name":"root","java.vm.name":"Java HotSpot(TM) 64-Bit Server VM","sun.java.command":"org.apache.spark.deploy.SparkSubmit --master spark://master:7077 --conf spark.driver.memory=512m --class org.apache.spark.examples.SparkPi --num-executors 1 --executor-memory 512m --executor-cores 1 /home/spark-3.0.1/examples/jars/spark-examples_2.12-3.0.1.jar","java.home":"/home/java/jre","java.version":"1.8.0_162","sun.io.unicode.encoding":"UnicodeLittle"},"Classpath Entries":{"/home/spark-3.0.1/jars/jaxb-runtime-2.3.2.jar":"System Classpath","/home/spark-3.0.1/jars/scala-collection-compat_2.12-2.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/spire-platform_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/libfb303-0.9.3.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-mapreduce-client-jobclient-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/arpack_combined_all-0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-graphx_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-container-servlet-core-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/htrace-core4-4.1.0-incubating.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-format-2.4.0.jar":"System Classpath","/home/spark-3.0.1/jars/logging-interceptor-3.12.6.jar":"System Classpath","/home/spark-3.0.1/jars/breeze_2.12-1.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-admin-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/netty-all-4.1.47.Final.jar":"System Classpath","/home/spark-3.0.1/jars/istack-commons-runtime-3.0.8.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-crypto-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-sketch_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/avro-mapred-1.8.2-hadoop2.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/joda-time-2.10.5.jar":"System Classpath","/home/spark-3.0.1/jars/jpam-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/javax.inject-1.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-graphite-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/chill-java-0.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/okio-1.15.0.jar":"System Classpath","/home/spark-3.0.1/jars/kryo-shaded-4.0.2.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-ast_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/antlr4-runtime-4.7.1.jar":"System Classpath","spark://master:46515/jars/spark-examples_2.12-3.0.1.jar":"Added By User","/home/spark-3.0.1/jars/guice-4.0.jar":"System Classpath","/home/spark-3.0.1/jars/core-1.1.2.jar":"System Classpath","/home/spark-3.0.1/jars/commons-text-1.6.jar":"System Classpath","/home/spark-3.0.1/jars/commons-pool-1.5.4.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-client-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-asn1-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-cli-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/lz4-java-1.7.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-mesos_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-simplekdc-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/breeze-macros_2.12-1.0.jar":"System Classpath","/home/spark-3.0.1/jars/dnsjava-2.1.7.jar":"System Classpath","/home/spark-3.0.1/jars/okhttp-2.7.5.jar":"System Classpath","/home/spark-3.0.1/jars/transaction-api-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/xbean-asm7-shaded-4.15.jar":"System Classpath","/home/spark-3.0.1/jars/spark-core_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-jvm-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/paranamer-2.8.jar":"System Classpath","/home/spark-3.0.1/jars/mesos-1.4.0-shaded-protobuf.jar":"System Classpath","/home/spark-3.0.1/jars/ST4-4.0.4.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-container-servlet-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/hive-vector-code-gen-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/derby-10.12.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/compress-lzf-1.0.3.jar":"System Classpath","/home/spark-3.0.1/jars/jaxb-api-2.2.11.jar":"System Classpath","/home/spark-3.0.1/jars/orc-shims-1.5.10.jar":"System Classpath","/home/spark-3.0.1/jars/jcl-over-slf4j-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/spark-kvstore_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hk2-locator-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-network-shuffle_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hk2-utils-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-server-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/kubernetes-model-4.9.2.jar":"System Classpath","/home/spark-3.0.1/jars/commons-lang3-3.9.jar":"System Classpath","/home/spark-3.0.1/jars/spark-mllib_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-common-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/guice-servlet-4.0.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-annotations-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/audience-annotations-0.5.0.jar":"System Classpath","/home/spark-3.0.1/jars/slf4j-api-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/spark-streaming_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-0.23-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/avro-ipc-1.8.2.jar":"System Classpath","/home/spark-3.0.1/jars/scala-xml_2.12-1.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-server-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/stax-api-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/gson-2.2.4.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-server-web-proxy-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-core-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/curator-framework-2.13.0.jar":"System Classpath","/home/spark-3.0.1/jars/stax2-api-3.1.4.jar":"System Classpath","/home/spark-3.0.1/jars/commons-cli-1.2.jar":"System Classpath","/home/spark-3.0.1/jars/commons-collections-3.2.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-core-2.10.0.jar":"System Classpath","/home/hadoop-3.2.1/etc/hadoop/":"System Classpath","/home/spark-3.0.1/jars/kerb-client-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/generex-1.0.2.jar":"System Classpath","/home/spark-3.0.1/jars/jsr305-3.0.0.jar":"System Classpath","/home/spark-3.0.1/jars/scala-parser-combinators_2.12-1.1.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-jaxrs-base-2.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/jta-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-column-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-module-scala_2.12-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/janino-3.0.16.jar":"System Classpath","/home/spark-3.0.1/jars/hive-llap-common-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/spark-sql_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/json-smart-2.3.jar":"System Classpath","/home/spark-3.0.1/jars/scala-library-2.12.10.jar":"System Classpath","/home/spark-3.0.1/jars/hive-metastore-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/log4j-1.2.17.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-encoding-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/xz-1.5.jar":"System Classpath","/home/spark-3.0.1/jars/aircompressor-0.10.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-scalap_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.inject-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-scheduler-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-annotations-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/threeten-extra-1.5.0.jar":"System Classpath","/home/spark-3.0.1/jars/super-csv-2.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/accessors-smart-1.2.jar":"System Classpath","/home/spark-3.0.1/jars/hive-jdbc-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/orc-mapreduce-1.5.10.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-media-jaxb-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/scala-compiler-2.12.10.jar":"System Classpath","/home/spark-3.0.1/jars/opencsv-2.3.jar":"System Classpath","/home/spark-3.0.1/conf/":"System Classpath","/home/spark-3.0.1/jars/commons-logging-1.1.3.jar":"System Classpath","/home/spark-3.0.1/jars/spark-repl_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-dataformat-yaml-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/automaton-1.11-8.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-mapper-asl-1.9.13.jar":"System Classpath","/home/spark-3.0.1/jars/shapeless_2.12-2.3.3.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-common-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/slf4j-log4j12-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/json-1.8.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-common-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-client-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-core-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/nimbus-jose-jwt-4.41.1.jar":"System Classpath","/home/spark-3.0.1/jars/HikariCP-2.5.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-client-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/ivy-2.4.0.jar":"System Classpath","/home/spark-3.0.1/jars/shims-0.7.45.jar":"System Classpath","/home/spark-3.0.1/jars/commons-lang-2.6.jar":"System Classpath","/home/spark-3.0.1/jars/commons-compress-1.8.1.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-registry-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/stream-2.9.6.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-datatype-jsr310-2.10.3.jar":"System Classpath","/home/spark-3.0.1/jars/javassist-3.25.0-GA.jar":"System Classpath","/home/spark-3.0.1/jars/commons-httpclient-3.1.jar":"System Classpath","/home/spark-3.0.1/jars/machinist_2.12-0.6.8.jar":"System Classpath","/home/spark-3.0.1/jars/spark-launcher_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/arrow-vector-0.15.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-hk2-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/hive-common-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/zstd-jni-1.4.4-3.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-json-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/token-provider-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/guava-14.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/zookeeper-3.4.14.jar":"System Classpath","/home/spark-3.0.1/jars/commons-net-3.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-catalyst_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-math3-3.4.1.jar":"System Classpath","/home/spark-3.0.1/jars/arrow-format-0.15.1.jar":"System Classpath","/home/spark-3.0.1/jars/univocity-parsers-2.9.0.jar":"System Classpath","/home/spark-3.0.1/jars/httpcore-4.4.12.jar":"System Classpath","/home/spark-3.0.1/jars/py4j-0.10.9.jar":"System Classpath","/home/spark-3.0.1/jars/zjsonpatch-0.3.0.jar":"System Classpath","/home/spark-3.0.1/jars/spire-macros_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-yarn_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/macro-compat_2.12-1.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/antlr-runtime-3.5.2.jar":"System Classpath","/home/spark-3.0.1/jars/jodd-core-3.5.2.jar":"System Classpath","/home/spark-3.0.1/jars/jdo-api-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/JLargeArrays-1.5.jar":"System Classpath","/home/spark-3.0.1/jars/hive-serde-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-core-asl-1.9.13.jar":"System Classpath","/home/spark-3.0.1/jars/spark-unsafe_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-jmx-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.xml.bind-api-2.3.2.jar":"System Classpath","/home/spark-3.0.1/jars/algebra_2.12-2.0.0-M2.jar":"System Classpath","/home/spark-3.0.1/jars/bonecp-0.8.0.RELEASE.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-core_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/datanucleus-core-4.1.17.jar":"System Classpath","/home/spark-3.0.1/jars/osgi-resource-locator-1.0.3.jar":"System Classpath","/home/spark-3.0.1/jars/hk2-api-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/cats-kernel_2.12-2.0.0-M4.jar":"System Classpath","/home/spark-3.0.1/jars/datanucleus-api-jdo-4.2.4.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.validation-api-2.0.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-jaxrs-json-provider-2.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/httpclient-4.5.6.jar":"System Classpath","/home/spark-3.0.1/jars/oro-2.0.8.jar":"System Classpath","/home/spark-3.0.1/jars/re2j-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-jackson-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-hdfs-client-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/kubernetes-client-4.9.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-module-paranamer-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/flatbuffers-java-1.9.0.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-jackson_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/minlog-1.3.0.jar":"System Classpath","/home/spark-3.0.1/jars/javax.jdo-3.2.0-m3.jar":"System Classpath","/home/spark-3.0.1/jars/commons-crypto-1.0.0.jar":"System Classpath","/home/spark-3.0.1/jars/spark-tags_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/velocity-1.5.jar":"System Classpath","/home/spark-3.0.1/jars/leveldbjni-all-1.8.jar":"System Classpath","/home/spark-3.0.1/jars/pyrolite-4.30.jar":"System Classpath","/home/spark-3.0.1/jars/chill_2.12-0.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/RoaringBitmap-0.7.45.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-databind-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/hive-beeline-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-auth-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/orc-core-1.5.10.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-api-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/curator-recipes-2.13.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-util-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-hadoop-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-dbcp-1.4.jar":"System Classpath","/home/spark-3.0.1/jars/libthrift-0.12.0.jar":"System Classpath","/home/spark-3.0.1/jars/scala-reflect-2.12.10.jar":"System Classpath","/home/spark-3.0.1/jars/okhttp-3.12.6.jar":"System Classpath","/home/spark-3.0.1/jars/protobuf-java-2.5.0.jar":"System Classpath","/home/spark-3.0.1/jars/javax.servlet-api-3.1.0.jar":"System Classpath","/home/spark-3.0.1/jars/geronimo-jcache_1.0_spec-1.0-alpha-1.jar":"System Classpath","/home/spark-3.0.1/jars/jsp-api-2.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-util-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-kubernetes_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-identity-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-mapreduce-client-core-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/jline-2.14.6.jar":"System Classpath","/home/spark-3.0.1/jars/spark-hive-thriftserver_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spire_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-module-jaxb-annotations-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.annotation-api-1.3.5.jar":"System Classpath","/home/spark-3.0.1/jars/activation-1.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-config-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/woodstox-core-5.0.3.jar":"System Classpath","/home/spark-3.0.1/jars/snakeyaml-1.24.jar":"System Classpath","/home/spark-3.0.1/jars/spark-hive_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-server-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/arrow-memory-0.15.1.jar":"System Classpath","/home/spark-3.0.1/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar":"System Classpath","/home/spark-3.0.1/jars/avro-1.8.2.jar":"System Classpath","/home/spark-3.0.1/jars/spark-tags_2.12-3.0.1-tests.jar":"System Classpath","/home/spark-3.0.1/jars/aopalliance-1.0.jar":"System Classpath","/home/spark-3.0.1/jars/jcip-annotations-1.0-1.jar":"System Classpath","/home/spark-3.0.1/jars/curator-client-2.13.0.jar":"System Classpath","/home/spark-3.0.1/jars/aopalliance-repackaged-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-common-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/hive-exec-2.3.7-core.jar":"System Classpath","/home/spark-3.0.1/jars/jul-to-slf4j-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/javolution-5.5.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-network-common_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/JTransforms-3.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-beanutils-1.9.4.jar":"System Classpath","/home/spark-3.0.1/jars/datanucleus-rdbms-4.1.19.jar":"System Classpath","/home/spark-3.0.1/jars/commons-io-2.4.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-xdr-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-daemon-1.0.13.jar":"System Classpath","/home/spark-3.0.1/jars/kubernetes-model-common-4.9.2.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-mapreduce-client-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/commons-codec-1.10.jar":"System Classpath","/home/spark-3.0.1/jars/commons-compiler-3.0.16.jar":"System Classpath","/home/spark-3.0.1/jars/objenesis-2.5.1.jar":"System Classpath","/home/spark-3.0.1/jars/spire-util_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/ehcache-3.3.1.jar":"System Classpath","/home/spark-3.0.1/jars/snappy-java-1.1.7.5.jar":"System Classpath","/home/spark-3.0.1/jars/hive-storage-api-2.7.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-pkix-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.activation-api-1.2.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-mllib-local_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.ws.rs-api-2.1.6.jar":"System Classpath","/home/spark-3.0.1/jars/commons-configuration2-2.1.1.jar":"System Classpath"}}
{"Event":"SparkListenerApplicationStart","App Name":"Spark Pi","App ID":"app-20201019233352-0002","Timestamp":1603121631550,"User":"root"}
{"Event":"SparkListenerJobStart","Job ID":0,"Submission Time":1603121635939,"Stage Infos":[{"Stage ID":0,"Stage Attempt ID":0,"Stage Name":"reduce at SparkPi.scala:38","Number of Tasks":2,"RDD Info":[{"RDD ID":1,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"map\"}","Callsite":"map at SparkPi.scala:34","Parent IDs":[0],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0},{"RDD ID":0,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"0\",\"name\":\"parallelize\"}","Callsite":"parallelize at SparkPi.scala:34","Parent IDs":[],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0}],"Parent IDs":[],"Details":"org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)\norg.apache.spark.examples.SparkPi$.main(SparkPi.scala:38)\norg.apache.spark.examples.SparkPi.main(SparkPi.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)\norg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)\norg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)\norg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)\norg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","Accumulables":[]}],"Stage IDs":[0],"Properties":{"spark.rdd.scope":"{\"id\":\"2\",\"name\":\"reduce\"}","spark.rdd.scope.noOverride":"true"}}
{"Event":"SparkListenerStageSubmitted","Stage Info":{"Stage ID":0,"Stage Attempt ID":0,"Stage Name":"reduce at SparkPi.scala:38","Number of Tasks":2,"RDD Info":[{"RDD ID":1,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"map\"}","Callsite":"map at SparkPi.scala:34","Parent IDs":[0],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0},{"RDD ID":0,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"0\",\"name\":\"parallelize\"}","Callsite":"parallelize at SparkPi.scala:34","Parent IDs":[],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0}],"Parent IDs":[],"Details":"org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)\norg.apache.spark.examples.SparkPi$.main(SparkPi.scala:38)\norg.apache.spark.examples.SparkPi.main(SparkPi.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)\norg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)\norg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)\norg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)\norg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","Submission Time":1603121636405,"Accumulables":[]},"Properties":{"spark.rdd.scope":"{\"id\":\"2\",\"name\":\"reduce\"}","spark.rdd.scope.noOverride":"true"}}
{"Event":"SparkListenerExecutorAdded","Timestamp":1603121637799,"Executor ID":"0","Executor Info":{"Host":"192.168.153.135","Total Cores":1,"Log Urls":{"stdout":"http://192.168.153.135:8081/logPage/?appId=app-20201019233352-0002&executorId=0&logType=stdout","stderr":"http://192.168.153.135:8081/logPage/?appId=app-20201019233352-0002&executorId=0&logType=stderr"},"Attributes":{},"Resources":{}}}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"0","Host":"192.168.153.135","Port":34409},"Maximum Memory":97832140,"Timestamp":1603121637949,"Maximum Onheap Memory":97832140,"Maximum Offheap Memory":0}
{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task Info":{"Task ID":0,"Index":0,"Attempt":0,"Launch Time":1603121638008,"Executor ID":"0","Host":"192.168.153.135","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
{"Event":"SparkListenerExecutorAdded","Timestamp":1603121638131,"Executor ID":"2","Executor Info":{"Host":"192.168.153.137","Total Cores":1,"Log Urls":{"stdout":"http://192.168.153.137:8081/logPage/?appId=app-20201019233352-0002&executorId=2&logType=stdout","stderr":"http://192.168.153.137:8081/logPage/?appId=app-20201019233352-0002&executorId=2&logType=stderr"},"Attributes":{},"Resources":{}}}
{"Event":"SparkListenerExecutorAdded","Timestamp":1603121638228,"Executor ID":"1","Executor Info":{"Host":"192.168.153.136","Total Cores":1,"Log Urls":{"stdout":"http://192.168.153.136:8082/logPage/?appId=app-20201019233352-0002&executorId=1&logType=stdout","stderr":"http://192.168.153.136:8082/logPage/?appId=app-20201019233352-0002&executorId=1&logType=stderr"},"Attributes":{},"Resources":{}}}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"2","Host":"192.168.153.137","Port":39783},"Maximum Memory":122644070,"Timestamp":1603121638354,"Maximum Onheap Memory":122644070,"Maximum Offheap Memory":0}
{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task Info":{"Task ID":1,"Index":1,"Attempt":0,"Launch Time":1603121638428,"Executor ID":"2","Host":"192.168.153.137","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"1","Host":"192.168.153.136","Port":44998},"Maximum Memory":122644070,"Timestamp":1603121638496,"Maximum Onheap Memory":122644070,"Maximum Offheap Memory":0}
{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":0,"Index":0,"Attempt":0,"Launch Time":1603121638008,"Executor ID":"0","Host":"192.168.153.135","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1603121639525,"Failed":false,"Killed":false,"Accumulables":[{"ID":5,"Name":"internal.metrics.jvmGCTime","Update":59,"Value":59,"Internal":true,"Count Failed Values":true},{"ID":4,"Name":"internal.metrics.resultSize","Update":842,"Value":842,"Internal":true,"Count Failed Values":true},{"ID":3,"Name":"internal.metrics.executorCpuTime","Update":66570998,"Value":66570998,"Internal":true,"Count Failed Values":true},{"ID":2,"Name":"internal.metrics.executorRunTime","Update":70,"Value":70,"Internal":true,"Count Failed Values":true},{"ID":1,"Name":"internal.metrics.executorDeserializeCpuTime","Update":1104556264,"Value":1104556264,"Internal":true,"Count Failed Values":true},{"ID":0,"Name":"internal.metrics.executorDeserializeTime","Update":1361,"Value":1361,"Internal":true,"Count Failed Values":true}]},"Task Executor Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task Metrics":{"Executor Deserialize Time":1361,"Executor Deserialize CPU Time":1104556264,"Executor Run Time":70,"Executor CPU Time":66570998,"Peak Execution Memory":0,"Result Size":842,"JVM GC Time":59,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated Blocks":[]}}
{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":1,"Index":1,"Attempt":0,"Launch Time":1603121638428,"Executor ID":"2","Host":"192.168.153.137","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1603121640774,"Failed":false,"Killed":false,"Accumulables":[{"ID":5,"Name":"internal.metrics.jvmGCTime","Update":69,"Value":128,"Internal":true,"Count Failed Values":true},{"ID":4,"Name":"internal.metrics.resultSize","Update":842,"Value":1684,"Internal":true,"Count Failed Values":true},{"ID":3,"Name":"internal.metrics.executorCpuTime","Update":70763236,"Value":137334234,"Internal":true,"Count Failed Values":true},{"ID":2,"Name":"internal.metrics.executorRunTime","Update":160,"Value":230,"Internal":true,"Count Failed Values":true},{"ID":1,"Name":"internal.metrics.executorDeserializeCpuTime","Update":1007597102,"Value":2112153366,"Internal":true,"Count Failed Values":true},{"ID":0,"Name":"internal.metrics.executorDeserializeTime","Update":2121,"Value":3482,"Internal":true,"Count Failed Values":true}]},"Task Executor Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task Metrics":{"Executor Deserialize Time":2121,"Executor Deserialize CPU Time":1007597102,"Executor Run Time":160,"Executor CPU Time":70763236,"Peak Execution Memory":0,"Result Size":842,"JVM GC Time":69,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated Blocks":[]}}
{"Event":"SparkListenerStageCompleted","Stage Info":{"Stage ID":0,"Stage Attempt ID":0,"Stage Name":"reduce at SparkPi.scala:38","Number of Tasks":2,"RDD Info":[{"RDD ID":1,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"map\"}","Callsite":"map at SparkPi.scala:34","Parent IDs":[0],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0},{"RDD ID":0,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"0\",\"name\":\"parallelize\"}","Callsite":"parallelize at SparkPi.scala:34","Parent IDs":[],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0}],"Parent IDs":[],"Details":"org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)\norg.apache.spark.examples.SparkPi$.main(SparkPi.scala:38)\norg.apache.spark.examples.SparkPi.main(SparkPi.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)\norg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)\norg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)\norg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)\norg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","Submission Time":1603121636405,"Completion Time":1603121640779,"Accumulables":[{"ID":2,"Name":"internal.metrics.executorRunTime","Value":230,"Internal":true,"Count Failed Values":true},{"ID":5,"Name":"internal.metrics.jvmGCTime","Value":128,"Internal":true,"Count Failed Values":true},{"ID":4,"Name":"internal.metrics.resultSize","Value":1684,"Internal":true,"Count Failed Values":true},{"ID":1,"Name":"internal.metrics.executorDeserializeCpuTime","Value":2112153366,"Internal":true,"Count Failed Values":true},{"ID":3,"Name":"internal.metrics.executorCpuTime","Value":137334234,"Internal":true,"Count Failed Values":true},{"ID":0,"Name":"internal.metrics.executorDeserializeTime","Value":3482,"Internal":true,"Count Failed Values":true}]}}
{"Event":"SparkListenerJobEnd","Job ID":0,"Completion Time":1603121640792,"Job Result":{"Result":"JobSucceeded"}}
{"Event":"SparkListenerApplicationEnd","Timestamp":1603121640817}

    一个日志文件代表了一个Spark App，日志中每行为JSON格式的Spark App事件内容，文件由上而下顺序的输出了整个Spark App执行的事件过程。

事件分析

    利用事件日志提供的多个指标，我们可以发现一些Spark App中存在的程序错误和性能故障，大概有如下分析方法：

  1. 获取Task执行结果，对失败的Task进行异常总结，针对异常形成统一解决方案。
  2. 比较Task的平均执行时间和最大执行时间，以确定是否有某个Task拖慢整体任务进度。
  3. 观察Task的平均数据量大小和最大数据量大小，以确定是否存在数据倾斜。
  4. 分析Task的JVM GC时间，观察是否有GC时间过慢的情况，主要观察GC占用CPU时间的百分比，如果占用过多则需要对GC进行优化。

你可能感兴趣的:(大数据技术,spark,flink,event)

  • 【加密社】Solidity 中的事件机制及其应用 加密社 闲侃区块链智能合约区块链
    加密社引言在Solidity合约开发过程中，事件（Events）是一种非常重要的机制。它们不仅能够让开发者记录智能合约的重要状态变更，还能够让外部系统（如前端应用）监听这些状态的变化。本文将详细介绍Solidity中的事件机制以及如何利用不同的手段来触发、监听和获取这些事件。事件存储的地方当我们在Solidity合约中使用emit关键字触发事件时，该事件会被记录在区块链的交易收据中。具体而言，事件
  • nosql数据库技术与应用知识点 皆过客，揽星河 NoSQLnosql数据库大数据数据分析数据结构非关系型数据库
    Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
  • 【ARM Cortex-M 系列 2.3 -- Cortex-M7 Debug event 详细介绍】 主公讲 ARM #ARM系列arm开发debugevent
    请阅读【嵌入式开发学习必备专栏】文章目录Cortex-M7DebugeventDebugeventsCortex-M7Debugevent在ARMCortex-M7架构中，调试事件（DebugEvent）是由于调试原因而触发的事件。一个调试事件会导致以下几种情况之一发生：进入调试状态：如果启用了停滞调试（HaltingDebug），一个调试事件会使处理器在调试状态下停滞。通过将DHCSR.C_DE
  • 如何利用大数据与AI技术革新相亲交友体验 h17711347205 回归算法安全系统架构交友小程序
    在数字化时代，大数据和人工智能（AI）技术正逐渐革新相亲交友体验，为寻找爱情的过程带来前所未有的变革（编辑h17711347205）。通过精准分析和智能匹配，这些技术能够极大地提高相亲交友系统的效率和用户体验。大数据的力量大数据技术能够收集和分析用户的行为模式、偏好和互动数据，为相亲交友系统提供丰富的信息资源。通过分析用户的搜索历史、浏览记录和点击行为，系统能够深入了解用户的兴趣和需求，从而提供更
  • Android实现监听事件的方法 Amy木婉清
    1.通过内部类实现2.通过匿名内部类实现3.通过事件源所在类实现4.通过外部类实现5.布局文件中onclick属性(针对点击事件)1.通过内部类实现代码:privateButtonmBtnEvent;//oncreate中mBtnEvent.setOnClickListener(newOnClick());//内部类实现监听classOnClickimplementsView.OnClickLis
  • 未来软件市场是怎么样的？做开发的生存空间如何？ cesske 软件需求
    目录前言一、未来软件市场的发展趋势二、软件开发人员的生存空间前言未来软件市场是怎么样的？做开发的生存空间如何？一、未来软件市场的发展趋势技术趋势：人工智能与机器学习：随着技术的不断成熟，人工智能将在更多领域得到应用，如智能客服、自动驾驶、智能制造等，这将极大地推动软件市场的增长。云计算与大数据：云计算服务将继续普及，大数据技术的应用也将更加广泛。企业将更加依赖云计算和大数据来优化运营、提升效率，并
  • 创建一个完整的购物商城系统是一个复杂的项目，涉及前端（用户界面）、后端（服务器逻辑）、数据库等多个部分。由于篇幅限制，我无法在这里提供一个完整的系统代码，但我可以分别给出一些关键部分的示例代码，涵盖几 uthRaman 前端ui服务器
    前端（HTML/CSS/JavaScript）grsyzp.cnHTML页面结构（index.html）html购物商城欢迎来到购物商城JavaScript（Ajax请求商品数据，app.js）javascriptdocument.addEventListener('DOMContentLoaded',function(){fetch('/api/products').then(response=
  • Kafka详细解析与应用分析 芊言芊语 kafka分布式
    Kafka是一个开源的分布式事件流平台（EventStreamingPlatform），由LinkedIn公司最初采用Scala语言开发，并基于ZooKeeper协调管理。如今，Kafka已经被Apache基金会纳入其项目体系，广泛应用于大数据实时处理领域。Kafka凭借其高吞吐量、持久化、分布式和可靠性的特点，成为构建实时流数据管道和流处理应用程序的重要工具。Kafka架构Kafka的架构主要由
  • 分享一个基于python的电子书数据采集与可视化分析 hadoop电子书数据分析与推荐系统 spark大数据毕设项目（源码、调试、LW、开题、PPT) 计算机源码社 Python项目大数据大数据pythonhadoop计算机毕业设计选题计算机毕业设计源码数据分析spark毕设
    作者：计算机源码社个人简介：本人八年开发经验，擅长Java、Python、PHP、.NET、Node.js、Android、微信小程序、爬虫、大数据、机器学习等，大家有这一块的问题可以一起交流！学习资料、程序开发、技术解答、文档报告如需要源码，可以扫取文章下方二维码联系咨询Java项目微信小程序项目Android项目Python项目PHP项目ASP.NET项目Node.js项目选题推荐项目实战|p
  • Spark 组件 GraphX、Streaming 叶域 大数据sparkspark大数据分布式
    Spark组件GraphX、Streaming一、SparkGraphX1.1GraphX的主要概念1.2GraphX的核心操作1.3示例代码1.4GraphX的应用场景二、SparkStreaming2.1SparkStreaming的主要概念2.2示例代码2.3SparkStreaming的集成2.4SparkStreaming的应用场景SparkGraphX用于处理图和图并行计算。Graph
  • 关于django中几个重要的gunicorn worker的配置 给我起把狙 djangogunicorn
    关于django中几个重要的gunicornworker的配置一、worker_classworker_class是Gunicorn的配置参数之一，它指定了工作进程（worker）的类型。不同的worker_class提供了不同的并发模型，适合不同类型的应用场景。sync和gevent是两种常见的worker_class，它们的作用和区别如下：1.sync（同步worker）默认值：如果没有指定w
  • 全面指南：用户行为从前端数据采集到实时处理的最佳实践 数字沉思 营销流量运营系统架构前端内容运营大数据
    引言在当今的数据驱动世界，实时数据采集和处理已经成为企业做出及时决策的重要手段。本文将详细介绍如何通过前端JavaScript代码采集用户行为数据、利用API和Kafka进行数据传输、通过Flink实时处理数据的完整流程。无论你是想提升产品体验还是做用户行为分析，这篇文章都将为你提供全面的解决方案。设计一个通用的ClickHouse表来存储用户事件时，需要考虑多种因素，包括事件类型、时间戳、用户信
  • bat批处理循环执行adb命令 erhtre 程序员adb
    ping-n2127.0>nul等待2s:one::adbshellinputswipe90010007001000adbshellinputkeyevent25%KEYCODE_VOLUME_DOWN%ping-n2127.0>nulgototwo:twoadbshellinputkeyevent25%KEYCODE_VOLUME_DOWN%ping-n3127.0>nulgotoone附：An
  • 大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏 租房推荐系统 58同城租房爬虫 房源推荐系统 房价预测系统 计算机毕业设计 机器学习 深度学习 人工智能 2401_84572577 程序员大数据hadoop人工智能
    做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
  • 架构评审的自动化与人工智能: 如何提高效率 光剑书架上的书 架构自动化人工智能运维
    1.背景介绍架构评审是软件开发过程中的一个关键环节，它旨在确保软件架构的质量、可维护性和可扩展性。传统的架构评审通常是由人工进行，需要大量的时间和精力。随着大数据技术和人工智能的发展，自动化和人工智能技术已经开始应用于架构评审，从而提高评审的效率和准确性。在本文中，我们将讨论如何通过自动化和人工智能技术来提高架构评审的效率。我们将从以下几个方面进行讨论：背景介绍核心概念与联系核心算法原理和具体操作
  • 详解 Flink 的常见部署方式 文刀小桂 Flinkflink大数据
    一、常见部署模式分类1.按是否依赖外部资源调度1.1Standalone模式独立模式(Standalone)是独立运行的，不依赖任何外部的资源管理平台，只需要运行所有Flink组件服务1.2Yarn模式Yarn模式是指客户端把Flink应用提交给Yarn的ResourceManager,Yarn的ResourceManager会在Yarn的NodeManager上创建容器。在这些容器上，Flink
  • Mac 技术篇-应用程序被锁定无法进行卸载问题解决方法，文件、文件夹被锁定无法移入废纸篓处理方法 lq9527_ Mac使用macos
    在卸载Karabiner-Elements和Karabiner-EventViewer软件时，提示应用锁定，无法卸载。参照方法。在进行/bin/ls-dleO@App路径操作后，返回提示信息与链接方法略有区别。/bin/ls-dleO@App路径drwxr-xr-x@3root wheel uchg96 3 1 2022/Applications/Karabiner-Elements.appcom
  • 微信分享到朋友圈,怎么自定义分享的标题,图片,内容? 八分造作
    vartit=$('#wx-title').val();//标题varimg=$('#wx-img').val();//图片varcon=$('#wx-con').val();//简介varlink=$('#wx-link').val();//链接document.addEventListener('WeixinJSBridgeReady',functiononBridgeReady(){//发送
  • Spark集群的三种模式 MelodyYN #Sparksparkhadoopbigdata
    文章目录1、Spark的由来1.1Hadoop的发展1.2MapReduce与Spark对比2、Spark内置模块3、Spark运行模式3.1Standalone模式部署配置历史服务器配置高可用运行模式3.2Yarn模式安装部署配置历史服务器运行模式4、WordCount案例1、Spark的由来定义：Hadoop主要解决，海量数据的存储和海量数据的分析计算。Spark是一种基于内存的快速、通用、可
  • 分布式框架Celery七(Django-Celery-Flower实现异步和定时爬虫及其监控邮件告警) yjjpp2301 Celery分布式djangopython后端
    Django中集成方式安装模块pipinstallDjango==3.2.22pipinstallcelerypipinstallredispipinstalleventlet#在windows环境下需要安装eventlet包-----------pipinstalldjango-celery-beatpipinstalldjango-celery-resultspipinstalldjango-
  • Jetpack LiveData源码分析 xiangxiongfly915 #AndroidJetpack系列JetpackLiveData源码分析
    文章目录JetpackLiveData源码分析前提源码分析注册订阅流程LiveData#observe()LifecycleRegistry#addObserver()ObserverWithState#dispatchEvent()LiveData#removeObserver()LiveData$LifecycleBoundObserver类LifecycleBoundObserver#sho
  • 大数据之flink与hive 星辰_mya 大数据flinkhive
    其实吧我不太想写flink，因为线上经验确实不多，这也是我需要补的地方，没有条件创造条件，先来一篇吧flink：高性能低延迟流批一体的分布式计算框架基于事件时间对实时数据精准处理快速响应支持批处理，高效离线分析和数据挖掘数据仓库的引擎丰富数据源/接收器，集成多种数据存储格式和源，比较常见就是咱们今天的主题hive了checkpoint恢复机制，故障恢复快速恢复计算任务分布式弹性扩展，据业务灵活增加
  • Java中的大数据处理框架对比分析 省赚客app开发者 java开发语言
    Java中的大数据处理框架对比分析大家好，我是微赚淘客系统3.0的小编，是个冬天不穿秋裤，天冷也要风度的程序猿！今天，我们将深入探讨Java中常用的大数据处理框架，并对它们进行对比分析。大数据处理框架是现代数据驱动应用的核心，它们帮助企业处理和分析海量数据，以提取有价值的信息。本文将重点介绍ApacheHadoop、ApacheSpark、ApacheFlink和ApacheStorm这四种流行的
  • Tapable 信海利世 webpackwebpack前端
    一、Tapable是什么？Tapable是一个类似于Node.js的EventEmitter的库,主要是控制钩子函数的发布与订阅,控制着webpack的插件系统。Tapable库暴露了很多Hook（钩子）类，为插件提供挂载的钩子const{SyncHook,//同步钩子SyncBailHook,//同步熔断钩子即return一个非undefined的值，则不再继续执行后面的监听函数SyncWate
  • fetch-event-source 如何通过script全局引入 cuiyuchen111 javascript前端
    fetchEventSource源码中导出了两种类型的包cjs和esm。但是有个需求如何在原生是js中通过script标签引呢？需要加上type=module。今天介绍另一种方法下载源码文件：https://github.com/Azure/fetch-event-source.git安装:npminstall--save-devwebpackwebpack-clits-loadertypescr
  • 写出渗透测试信息收集详细流程 卿酌南烛_b805
    一、扫描域名漏洞：域名漏洞扫描工具有AWVS、APPSCAN、Netspark、WebInspect、Nmap、Nessus、天镜、明鉴、WVSS、RSAS等。二、子域名探测：1、dns域传送漏洞2、搜索引擎查找（通过Google、bing、搜索c段）3、通过ssl证书查询网站：https://myssl.com/ssl.html和https://www.chinassl.net/ssltools
  • 一文搞懂 Flink Task 数据交互之数据写源码 mn_kw flink交互java
    一文搞懂FlinkTask数据交互之数据写源码1.RecordWriterOutput2.RecordWriter3.数据分区器ChannelSelector4.数据输出模型ResultPartition5.子模型ResultSubpartition6.本地buffer池LocalBufferPool7.获取buffer8.将buffer添加到ResultSubpartitionFlink重要源码
  • treeselect只选了分支节点全选_vue Treeselect 树形下拉框:获取选中节点的ids和lables操作... weixin_39637285
    API:https://vue-treeselect.js.org/#events1.ids:即value1.lable:需要用到方法：@select(node,instanceId)和@deselect(node,instanceId)v-model="DRHA_EFaultModeTree_value":multiple="true":options="DRHA_EFaultModeTree_
  • 11-js事件基础 echozzi JavaScript基础javascript前端开发语言
    文章目录一、事件概念二、事件绑定方式1.onxxx2.事件监听器addEventListener三、事件解绑方式方法1：覆盖原事件处理（置空）方法2：移除事件监听器removeEventListener四、事件派发方式方法1：onxxx方法2：dispatchEvent五、事件对象window.event六、点击事件的光标坐标点1.以页面为参考坐标page2.以可视窗口为参考client3.相对事
  • 《昇思 25 天学习打卡营第 25 天 | 基于 MindSpore 实现 BERT 对话情绪识别 》 Sam9029 Mindscope模型学习深度学习
    《昇思25天学习打卡营第25天|基于MindSpore实现BERT对话情绪识别》活动地址：https://xihe.mindspore.cn/events/mindspore-training-camp签名：Sam9029环境配置确保安装了正确版本的MindSpore和MindNLP库。!pipuninstallmindspore-y!pipinstall-ihttps://pypi.mirror
  • 设计模式介绍 tntxia 设计模式
    设计模式来源于土木工程师 克里斯托弗 亚历山大（http://en.wikipedia.org/wiki/Christopher_Alexander）的早期作品。他经常发表一些作品，内容是总结他在解决设计问题方面的经验，以及这些知识与城市和建筑模式之间有何关联。有一天，亚历山大突然发现，重复使用这些模式可以让某些设计构造取得我们期望的最佳效果。 亚历山大与萨拉-石川佳纯和穆雷 西乐弗斯坦合作
  • android高级组件使用(一) 百合不是茶 androidRatingBarSpinner
    1、自动完成文本框（AutoCompleteTextView） AutoCompleteTextView从EditText派生出来，实际上也是一个文本编辑框，但它比普通编辑框多一个功能：当用户输入一个字符后，自动完成文本框会显示一个下拉菜单，供用户从中选择，当用户选择某个菜单项之后，AutoCompleteTextView按用户选择自动填写该文本框。 使用AutoCompleteTex
  • [网络与通讯]路由器市场大有潜力可挖掘 comsci 网络
              如果国内的电子厂商和计算机设备厂商觉得手机市场已经有点饱和了,那么可以考虑一下交换机和路由器市场的进入问题.....        这方面的技术和知识,目前处在一个开放型的状态,有利于各类小型电子企业进入  &nbs
  • 自写简单Redis内存统计shell 商人shang Linux shell统计Redis内存
    #!/bin/bash address="192.168.150.128:6666,192.168.150.128:6666" hosts=(${address//,/ }) sfile="staticts.log" for hostitem in ${hosts[@]} do ipport=(${hostitem
  • 单例模式(饿汉 vs懒汉) oloz 单例模式
    package 单例模式; /* * 应用场景:保证在整个应用之中某个对象的实例只有一个 * 单例模式种的《 懒汉模式》 * */ public class Singleton { //01 将构造方法私有化，外界就无法用new Singleton()的方式获得实例 private Singleton(){}; //02 申明类得唯一实例 priva
  • springMvc json支持 杨白白 json springmvc
    1.Spring mvc处理json需要使用jackson的类库，因此需要先引入jackson包 2在spring mvc中解析输入为json格式的数据:使用@RequestBody来设置输入 @RequestMapping("helloJson") public @ResponseBody JsonTest helloJson() {
  • android播放，掃描添加本地音頻文件 小桔子
            最近幾乎沒有什麽事情，繼續鼓搗我的小東西。想在項目中加入一個簡易的音樂播放器功能，就像華為p6桌面上那麼大小的音樂播放器。用過天天動聽或者QQ音樂播放器的人都知道，可已通過本地掃描添加歌曲。不知道他們是怎麼實現的，我覺得應該掃描設備上的所有文件，過濾出音頻文件，每個文件實例化為一個實體，記錄文件名、路徑、歌手、類型、大小等信息。具體算法思想，
  • oracle常用命令 aichenglong oracledba常用命令
    1 创建临时表空间 create temporary tablespace user_temp  tempfile 'D:\oracle\oradata\Oracle9i\user_temp.dbf' size 50m  autoextend on  next 50m maxsize 20480m  extent management local
  • 25个Eclipse插件 AILIKES eclipse插件
    提高代码质量的插件1. FindBugsFindBugs可以帮你找到Java代码中的bug，它使用Lesser GNU Public License的自由软件许可。2. CheckstyleCheckstyle插件可以集成到Eclipse IDE中去，能确保Java代码遵循标准代码样式。3. ECLemmaECLemma是一款拥有Eclipse Public License许可的免费工具，它提供了
  • Spring MVC拦截器+注解方式实现防止表单重复提交 baalwolf spring mvc
    原理：在新建页面中Session保存token随机码，当保存时验证，通过后删除，当再次点击保存时由于服务器端的Session中已经不存在了，所有无法验证通过。   1.新建注解：   ? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
  • 《Javascript高级程序设计(第3版)》闭包理解 bijian1013 JavaScript
    “闭包是指有权访问另一个函数作用域中的变量的函数。”--《Javascript高级程序设计(第3版)》         看以下代码： <script type="text/javascript"> function outer() { var i = 10; return f
  • AngularJS Module类的方法 bijian1013 JavaScriptAngularJSModule
            AngularJS中的Module类负责定义应用如何启动，它还可以通过声明的方式定义应用中的各个片段。我们来看看它是如何实现这些功能的。 一.Main方法在哪里         如果你是从Java或者Python编程语言转过来的，那么你可能很想知道AngularJS里面的main方法在哪里？这个把所
  • [Maven学习笔记七]Maven插件和目标 bit1129 maven插件
    插件(plugin)和目标(goal) Maven，就其本质而言，是一个插件执行框架，Maven的每个目标的执行逻辑都是由插件来完成的，一个插件可以有1个或者几个目标，比如maven-compiler-plugin插件包含compile和testCompile，即maven-compiler-plugin提供了源代码编译和测试源代码编译的两个目标   使用插件和目标使得我们可以干预
  • 【Hadoop八】Yarn的资源调度策略 bit1129 hadoop
    1. Hadoop的三种调度策略 Hadoop提供了3中作业调用的策略， FIFO Scheduler Fair Scheduler Capacity Scheduler 以上三种调度算法，在Hadoop MR1中就引入了，在Yarn中对它们进行了改进和完善.Fair和Capacity Scheduler用于多用户共享的资源调度   2. 多用户资源共享的调度
  • Nginx使用Linux内存加速静态文件访问 ronin47
    Nginx是一个非常出色的静态资源web服务器。如果你嫌它还不够快，可以把放在磁盘中的文件，映射到内存中，减少高并发下的磁盘IO。 先做几个假设。nginx.conf中所配置站点的路径是/home/wwwroot/res，站点所对应文件原始存储路径：/opt/web/res shell脚本非常简单，思路就是拷贝资源文件到内存中，然后在把网站的静态文件链接指向到内存中即可。具体如下：
  • 关于Unity3D中的Shader的知识 brotherlamp unityunity资料unity教程unity视频unity自学
    首先先解释下Unity3D的Shader，Unity里面的Shaders是使用一种叫ShaderLab的语言编写的，它同微软的FX文件或者NVIDIA的CgFX有些类似。传统意义上的vertex shader和pixel shader还是使用标准的Cg/HLSL 编程语言编写的。因此Unity文档里面的Shader，都是指用ShaderLab编写的代码，然后我们来看下Unity3D自带的60多个S
  • CopyOnWriteArrayList vs ArrayList bylijinnan java
    package com.ljn.base; import java.util.ArrayList; import java.util.Iterator; import java.util.List; import java.util.concurrent.CopyOnWriteArrayList; /** * 总述： * 1.ArrayListi不是线程安全的，CopyO
  • 内存中栈和堆的区别 chicony 内存
    1、内存分配方面：     堆：一般由程序员分配释放， 若程序员不释放，程序结束时可能由OS回收 。注意它与数据结构中的堆是两回事，分配方式是类似于链表。可能用到的关键字如下：new、malloc、delete、free等等。     栈：由编译器(Compiler)自动分配释放，存放函数的参数值，局部变量的值等。其操作方式类似于数据结构中
  • 回答一位网友对Scala的提问 chenchao051 scalamap
    本来准备在私信里直接回复了，但是发现不太方便，就简要回答在这里。 问题 写道 对于scala的简洁十分佩服，但又觉得比较晦涩，例如一例，Map("a" -> List(11,111)).flatMap(_._2)，可否说下最后那个函数做了什么，真正在开发的时候也会如此简洁？谢谢    先回答一点，在实际使用中，Scala毫无疑问就是这么简单。
  • mysql 取每组前几条记录 daizj mysql分组最大值最小值每组三条记录
    一、对分组的记录取前N条记录：例如：取每组的前3条最大的记录 1.用子查询： SELECT * FROM tableName a  WHERE 3> (SELECT COUNT(*) FROM  tableName b WHERE b.id=a.id AND b.cnt>a. cnt) ORDER BY a.id,a.account DE
  • HTTP深入浅出 http请求 dcj3sjt126com http
      HTTP(HyperText Transfer Protocol)是一套计算机通过网络进行通信的规则。计算机专家设计出HTTP，使HTTP客户（如Web浏览器）能够从HTTP服务器(Web服务器)请求信息和服务，HTTP目前协议的版本是1.1.HTTP是一种无状态的协议，无状态是指Web浏览器和Web服务器之间不需要建立持久的连接，这意味着当一个客户端向服务器端发出请求，然后We
  • 判断MySQL记录是否存在方法比较 dcj3sjt126com mysql
    把数据写入到数据库的时，常常会碰到先要检测要插入的记录是否存在，然后决定是否要写入。 　　我这里总结了判断记录是否存在的常用方法： 　　sql语句： select   count ( * )  from  tablename; 　　然后读取count(*)的值判断记录是否存在。对于这种方法性能上有些浪费，我们只是想判断记录记录是否存在，没有必要全部都查出来。
  • 对HTML XML的一点认识 e200702084 htmlxml
    感谢http://www.w3school.com.cn提供的资料 HTML 文档中的每个成分都是一个节点。 节点 根据 DOM，HTML 文档中的每个成分都是一个节点。 DOM 是这样规定的： 整个文档是一个文档节点 每个 HTML 标签是一个元素节点 包含在 HTML 元素中的文本是文本节点 每一个 HTML 属性是一个属性节点 注释属于注释节点 Node 层次
  • jquery分页插件 genaiwei jqueryWeb前端分页插件
    //jquery页码控件// 创建一个闭包    (function($) {      // 插件的定义      $.fn.pageTool = function(options) {        var totalPa
  • Mybatis与Ibatis对照入门于学习 Josh_Persistence mybatisibatis区别联系
    一、为什么使用IBatis/Mybatis         对于从事 Java EE 的开发人员来说，iBatis 是一个再熟悉不过的持久层框架了，在 Hibernate、JPA 这样的一站式对象 / 关系映射（O/R Mapping）解决方案盛行之前，iBaits 基本是持久层框架的不二选择。即使在持久层框架层出不穷的今天，iBatis 凭借着易学易用、
  • C中怎样合理决定使用那种整数类型？ 秋风扫落叶 c数据类型
    如果需要大数值(大于32767或小于32767), 使用long 型。 否则, 如果空间很重要 (如有大数组或很多结构), 使用 short 型。 除此之外, 就使用 int 型。 如果严格定义的溢出特征很重要而负值无关紧要, 或者你希望在操作二进制位和字节时避免符号扩展的问题, 请使用对应的无符号类型。 但是, 要注意在表达式中混用有符号和无符号值的情况。    &nbs
  • maven问题 zhb8015 maven问题
      问题1： Eclipse 中 新建maven项目 无法添加src/main/java 问题    eclipse创建maevn web项目，在选择maven_archetype_web原型后，默认只有src/main/resources这个Source Floder。     按照maven目录结构，添加src/main/ja
  • (二)androidpn-server tomcat版源码解析之--push消息处理 spjich javaandrodipn推送
    在 (一)androidpn-server tomcat版源码解析之--项目启动这篇中，已经描述了整个推送服务器的启动过程，并且把握到了消息的入口即XmppIoHandler这个类，今天我将继续往下分析下面的核心代码，主要分为3大块，链接创建，消息的发送，链接关闭。 先贴一段XmppIoHandler的部分代码 /** * Invoked from an I/O proc
  • 用js中的formData类型解决ajax提交表单时文件不能被serialize方法序列化的问题 中华好儿孙 JavaScriptAjaxWeb上传文件FormData
    var formData = new FormData($("#inputFileForm")[0]); $.ajax({ type:'post', url:webRoot+"/electronicContractUrl/webapp/uploadfile", data:formData, async: false, ca
  • mybatis常用jdbcType数据类型 ysj5125094 mybatismapperjdbcType
      MyBatis 通过包含的jdbcType 类型 BIT         FLOAT      CHAR          
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他