Wang_Zhenwei

flume学习（七）、（八）：如何使用event header中的key值以及自定义source

1.如何使用event header中的key值？
2.如何部署扩展自定义的spooling directory source？

前面我们已经说到我们在header中添加了一个key为：flume.client.log4j.logger.source ，然后有两个应用程序，一个设置为app1,一个设置为app2。

现在有这么一个需求，要将app1的日志输出到hdfs://master68:8020/flume/events/app1目录下面，app2的日志输出到hdfs://master68:8020/flume/events/app2目录下面，未来也可能有更多的应用程序的日志输出，也即每个程序的日志输出到各自自己的目录下面。

有了前面的头信息之后，我们可以非常简单的实现这个需求：

tier1.sinks.sink1.hdfs.path=hdfs://master68:8020/flume/events/%{flume.client.log4j.logger.source}

只需要改一下这一行，用%{flume.client.log4j.logger.source}，来替换具体的app日志目录即可。

按照以往的惯例，还是需求驱动学习，有位网友在我的flume学习五中留言提了一个问题如下：
我想实现一个功能，就在读一个文件的时候，将文件的名字和文件生成的日期作为event的header传到hdfs上时，不同的event存到不同的目录下，如一个文件是a.log.2014-07-25在hdfs上是存到/a/2014-07-25目录下，a.log.2014-07-26存到/a/2014-07-26目录下，就是每个文件对应自己的目录，这个要怎么实现。

带着这个问题，我又重新翻看了官方的文档，发现一个spooling directory source跟这个需求稍微有点吻合：它监视指定的文件夹下面有没有写入新的文件，有的话，就会把该文件内容传递给sink,然后将该文件后缀标示为.complete，表示已处理。提供了参数可以将文件名和文件全路径名添加到event的header中去。

现有的功能不能满足我们的需求，但是至少提供了一个方向：它能将文件名放入header！
当时就在祈祷源代码不要太复杂，这样我们在这个地方稍微修改修改，把文件名拆分一下，然后再放入header，这样就完成了我们想要的功能了。

于是就打开了源代码，果然不复杂，代码结构非常清晰，按照我的思路，稍微改了一下，就实现了这个功能，主要修改了与spooling directory source代码相关的三个类，分别是：ReliableSpoolingFileEventExtReader，SpoolDirectorySourceConfigurationExtConstants，SpoolDirectoryExtSource（在原类名的基础上增加了：Ext）代码如下：

/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package com.besttone.flume;
import java.io.File;
import java.io.FileFilter;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.FlumeException;
import org.apache.flume.annotations.InterfaceAudience;
import org.apache.flume.annotations.InterfaceStability;
import org.apache.flume.client.avro.ReliableEventReader;
import org.apache.flume.serialization.DecodeErrorPolicy;
import org.apache.flume.serialization.DurablePositionTracker;
import org.apache.flume.serialization.EventDeserializer;
import org.apache.flume.serialization.EventDeserializerFactory;
import org.apache.flume.serialization.PositionTracker;
import org.apache.flume.serialization.ResettableFileInputStream;
import org.apache.flume.serialization.ResettableInputStream;
import org.apache.flume.tools.PlatformDetect;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.base.Charsets;
import com.google.common.base.Optional;
import com.google.common.base.Preconditions;
import com.google.common.io.Files;
/**
*
* A {@link ReliableEventReader} which reads log data from files stored in a
* spooling directory and renames each file once all of its data has been read
* (through {@link EventDeserializer#readEvent()} calls). The user must
* {@link #commit()} each read, to indicate that the lines have been fully
* processed.
*
* Read calls will return no data if there are no files left to read. This
* class, in general, is not thread safe.
*
*
* This reader assumes that files with unique file names are left in the
* spooling directory and not modified once they are placed there. Any user
* behavior which violates these assumptions, when detected, will result in a
* FlumeException being thrown.
*
*
* This class makes the following guarantees, if above assumptions are met:
*
- *
- Once a log file has been renamed with the {@link #completedSuffix}, all
- * of its records have been read through the
- * {@link EventDeserializer#readEvent()} function and {@link #commit()}ed at
- * least once.
- *
- All files in the spooling directory will eventually be opened and
- * delivered to a {@link #readEvents(int)} caller.
- *
*/
@InterfaceAudience.Private
@InterfaceStability.Evolving
public class ReliableSpoolingFileEventExtReader implements ReliableEventReader {
private static final Logger logger = LoggerFactory
.getLogger(ReliableSpoolingFileEventExtReader.class);
static final String metaFileName = ".flumespool-main.meta";
private final File spoolDirectory;
private final String completedSuffix;
private final String deserializerType;
private final Context deserializerContext;
private final Pattern ignorePattern;
private final File metaFile;
private final boolean annotateFileName;
private final boolean annotateBaseName;
private final String fileNameHeader;
private final String baseNameHeader;
// 添加参数开始
private final boolean annotateFileNameExtractor;
private final String fileNameExtractorHeader;
private final Pattern fileNameExtractorPattern;
private final boolean convertToTimestamp;
private final String dateTimeFormat;
private final boolean splitFileName;
private final String splitBy;
private final String splitBaseNameHeader;
// 添加参数结束
private final String deletePolicy;
private final Charset inputCharset;
private final DecodeErrorPolicy decodeErrorPolicy;
private Optional currentFile = Optional.absent();
/** Always contains the last file from which lines have been read. **/
private Optional lastFileRead = Optional.absent();
private boolean committed = true;
/**
* Create a ReliableSpoolingFileEventReader to watch the given directory.
*/
private ReliableSpoolingFileEventExtReader(File spoolDirectory,
String completedSuffix, String ignorePattern,
String trackerDirPath, boolean annotateFileName,
String fileNameHeader, boolean annotateBaseName,
String baseNameHeader, String deserializerType,
Context deserializerContext, String deletePolicy,
String inputCharset, DecodeErrorPolicy decodeErrorPolicy,
boolean annotateFileNameExtractor, String fileNameExtractorHeader,
String fileNameExtractorPattern, boolean convertToTimestamp,
String dateTimeFormat, boolean splitFileName, String splitBy,
String splitBaseNameHeader) throws IOException {
// Sanity checks
Preconditions.checkNotNull(spoolDirectory);
Preconditions.checkNotNull(completedSuffix);
Preconditions.checkNotNull(ignorePattern);
Preconditions.checkNotNull(trackerDirPath);
Preconditions.checkNotNull(deserializerType);
Preconditions.checkNotNull(deserializerContext);
Preconditions.checkNotNull(deletePolicy);
Preconditions.checkNotNull(inputCharset);
// validate delete policy
if (!deletePolicy.equalsIgnoreCase(DeletePolicy.NEVER.name())
&& !deletePolicy
.equalsIgnoreCase(DeletePolicy.IMMEDIATE.name())) {
throw new IllegalArgumentException("Delete policies other than "
+ "NEVER and IMMEDIATE are not yet supported");
}
if (logger.isDebugEnabled()) {
logger.debug("Initializing {} with directory={}, metaDir={}, "
+ "deserializer={}", new Object[] {
ReliableSpoolingFileEventExtReader.class.getSimpleName(),
spoolDirectory, trackerDirPath, deserializerType });
}
// Verify directory exists and is readable/writable
Preconditions
.checkState(
spoolDirectory.exists(),
"Directory does not exist: "
+ spoolDirectory.getAbsolutePath());
Preconditions.checkState(spoolDirectory.isDirectory(),
"Path is not a directory: " + spoolDirectory.getAbsolutePath());
// Do a canary test to make sure we have access to spooling directory
try {
File canary = File.createTempFile("flume-spooldir-perm-check-",
".canary", spoolDirectory);
Files.write("testing flume file permissions\n", canary,
Charsets.UTF_8);
List lines = Files.readLines(canary, Charsets.UTF_8);
Preconditions.checkState(!lines.isEmpty(), "Empty canary file %s",
canary);
if (!canary.delete()) {
throw new IOException("Unable to delete canary file " + canary);
}
logger.debug("Successfully created and deleted canary file: {}",
canary);
} catch (IOException e) {
throw new FlumeException("Unable to read and modify files"
+ " in the spooling directory: " + spoolDirectory, e);
}
this.spoolDirectory = spoolDirectory;
this.completedSuffix = completedSuffix;
this.deserializerType = deserializerType;
this.deserializerContext = deserializerContext;
this.annotateFileName = annotateFileName;
this.fileNameHeader = fileNameHeader;
this.annotateBaseName = annotateBaseName;
this.baseNameHeader = baseNameHeader;
this.ignorePattern = Pattern.compile(ignorePattern);
this.deletePolicy = deletePolicy;
this.inputCharset = Charset.forName(inputCharset);
this.decodeErrorPolicy = Preconditions.checkNotNull(decodeErrorPolicy);
// 增加代码开始
this.annotateFileNameExtractor = annotateFileNameExtractor;
this.fileNameExtractorHeader = fileNameExtractorHeader;
this.fileNameExtractorPattern = Pattern
.compile(fileNameExtractorPattern);
this.convertToTimestamp = convertToTimestamp;
this.dateTimeFormat = dateTimeFormat;
this.splitFileName = splitFileName;
this.splitBy = splitBy;
this.splitBaseNameHeader = splitBaseNameHeader;
// 增加代码结束
File trackerDirectory = new File(trackerDirPath);
// if relative path, treat as relative to spool directory
if (!trackerDirectory.isAbsolute()) {
trackerDirectory = new File(spoolDirectory, trackerDirPath);
}
// ensure that meta directory exists
if (!trackerDirectory.exists()) {
if (!trackerDirectory.mkdir()) {
throw new IOException(
"Unable to mkdir nonexistent meta directory "
+ trackerDirectory);
}
}
// ensure that the meta directory is a directory
if (!trackerDirectory.isDirectory()) {
throw new IOException("Specified meta directory is not a directory"
+ trackerDirectory);
}
this.metaFile = new File(trackerDirectory, metaFileName);
}
/**
* Return the filename which generated the data from the last successful
* {@link #readEvents(int)} call. Returns null if called before any file
* contents are read.
*/
public String getLastFileRead() {
if (!lastFileRead.isPresent()) {
return null;
}
return lastFileRead.get().getFile().getAbsolutePath();
}
// public interface
public Event readEvent() throws IOException {
List events = readEvents(1);
if (!events.isEmpty()) {
return events.get(0);
} else {
return null;
}
}
public List readEvents(int numEvents) throws IOException {
if (!committed) {
if (!currentFile.isPresent()) {
throw new IllegalStateException("File should not roll when "
+ "commit is outstanding.");
}
logger.info("Last read was never committed - resetting mark position.");
currentFile.get().getDeserializer().reset();
} else {
// Check if new files have arrived since last call
if (!currentFile.isPresent()) {
currentFile = getNextFile();
}
// Return empty list if no new files
if (!currentFile.isPresent()) {
return Collections.emptyList();
}
}
EventDeserializer des = currentFile.get().getDeserializer();
List events = des.readEvents(numEvents);
/*
* It's possible that the last read took us just up to a file boundary.
* If so, try to roll to the next file, if there is one.
*/
if (events.isEmpty()) {
retireCurrentFile();
currentFile = getNextFile();
if (!currentFile.isPresent()) {
return Collections.emptyList();
}
events = currentFile.get().getDeserializer().readEvents(numEvents);
}
if (annotateFileName) {
String filename = currentFile.get().getFile().getAbsolutePath();
for (Event event : events) {
event.getHeaders().put(fileNameHeader, filename);
}
}
if (annotateBaseName) {
String basename = currentFile.get().getFile().getName();
for (Event event : events) {
event.getHeaders().put(baseNameHeader, basename);
}
}
// 增加代码开始
// 按正则抽取文件名的内容
if (annotateFileNameExtractor) {
Matcher matcher = fileNameExtractorPattern.matcher(currentFile
.get().getFile().getName());
if (matcher.find()) {
String value = matcher.group();
if (convertToTimestamp) {
DateTimeFormatter formatter = DateTimeFormat
.forPattern(dateTimeFormat);
DateTime dateTime = formatter.parseDateTime(value);
value = Long.toString(dateTime.getMillis());
}
for (Event event : events) {
event.getHeaders().put(fileNameExtractorHeader, value);
}
}
}
// 按分隔符拆分文件名
if (splitFileName) {
String[] splits = currentFile.get().getFile().getName()
.split(splitBy);
for (Event event : events) {
for (int i = 0; i < splits.length; i++) {
event.getHeaders().put(splitBaseNameHeader + i, splits[i]);
}
}
}
// 增加代码结束
committed = false;
lastFileRead = currentFile;
return events;
}
@Override
public void close() throws IOException {
if (currentFile.isPresent()) {
currentFile.get().getDeserializer().close();
currentFile = Optional.absent();
}
}
/** Commit the last lines which were read. */
@Override
public void commit() throws IOException {
if (!committed && currentFile.isPresent()) {
currentFile.get().getDeserializer().mark();
committed = true;
}
}
/**
* Closes currentFile and attempt to rename it.
*
* If these operations fail in a way that may cause duplicate log entries,
* an error is logged but no exceptions are thrown. If these operations fail
* in a way that indicates potential misuse of the spooling directory, a
* FlumeException will be thrown.
*
* @throws FlumeException
* if files do not conform to spooling assumptions
*/
private void retireCurrentFile() throws IOException {
Preconditions.checkState(currentFile.isPresent());
File fileToRoll = new File(currentFile.get().getFile()
.getAbsolutePath());
currentFile.get().getDeserializer().close();
// Verify that spooling assumptions hold
if (fileToRoll.lastModified() != currentFile.get().getLastModified()) {
String message = "File has been modified since being read: "
+ fileToRoll;
throw new IllegalStateException(message);
}
if (fileToRoll.length() != currentFile.get().getLength()) {
String message = "File has changed size since being read: "
+ fileToRoll;
throw new IllegalStateException(message);
}
if (deletePolicy.equalsIgnoreCase(DeletePolicy.NEVER.name())) {
rollCurrentFile(fileToRoll);
} else if (deletePolicy.equalsIgnoreCase(DeletePolicy.IMMEDIATE.name())) {
deleteCurrentFile(fileToRoll);
} else {
// TODO: implement delay in the future
throw new IllegalArgumentException("Unsupported delete policy: "
+ deletePolicy);
}
}
/**
* Rename the given spooled file
*
* @param fileToRoll
* @throws IOException
*/
private void rollCurrentFile(File fileToRoll) throws IOException {
File dest = new File(fileToRoll.getPath() + completedSuffix);
logger.info("Preparing to move file {} to {}", fileToRoll, dest);
// Before renaming, check whether destination file name exists
if (dest.exists() && PlatformDetect.isWindows()) {
/*
* If we are here, it means the completed file already exists. In
* almost every case this means the user is violating an assumption
* of Flume (that log files are placed in the spooling directory
* with unique names). However, there is a corner case on Windows
* systems where the file was already rolled but the rename was not
* atomic. If that seems likely, we let it pass with only a warning.
*/
if (Files.equal(currentFile.get().getFile(), dest)) {
logger.warn("Completed file " + dest
+ " already exists, but files match, so continuing.");
boolean deleted = fileToRoll.delete();
if (!deleted) {
logger.error("Unable to delete file "
+ fileToRoll.getAbsolutePath()
+ ". It will likely be ingested another time.");
}
} else {
String message = "File name has been re-used with different"
+ " files. Spooling assumptions violated for " + dest;
throw new IllegalStateException(message);
}
// Dest file exists and not on windows
} else if (dest.exists()) {
String message = "File name has been re-used with different"
+ " files. Spooling assumptions violated for " + dest;
throw new IllegalStateException(message);
// Destination file does not already exist. We are good to go!
} else {
boolean renamed = fileToRoll.renameTo(dest);
if (renamed) {
logger.debug("Successfully rolled file {} to {}", fileToRoll,
dest);
// now we no longer need the meta file
deleteMetaFile();
} else {
/*
* If we are here then the file cannot be renamed for a reason
* other than that the destination file exists (actually, that
* remains possible w/ small probability due to TOC-TOU
* conditions).
*/
String message = "Unable to move "
+ fileToRoll
+ " to "
+ dest
+ ". This will likely cause duplicate events. Please verify that "
+ "flume has sufficient permissions to perform these operations.";
throw new FlumeException(message);
}
}
}
/**
* Delete the given spooled file
*
* @param fileToDelete
* @throws IOException
*/
private void deleteCurrentFile(File fileToDelete) throws IOException {
logger.info("Preparing to delete file {}", fileToDelete);
if (!fileToDelete.exists()) {
logger.warn("Unable to delete nonexistent file: {}", fileToDelete);
return;
}
if (!fileToDelete.delete()) {
throw new IOException("Unable to delete spool file: "
+ fileToDelete);
}
// now we no longer need the meta file
deleteMetaFile();
}
/**
* Find and open the oldest file in the chosen directory. If two or more
* files are equally old, the file name with lower lexicographical value is
* returned. If the directory is empty, this will return an absent option.
*/
private Optional getNextFile() {
/* Filter to exclude finished or hidden files */
FileFilter filter = new FileFilter() {
public boolean accept(File candidate) {
String fileName = candidate.getName();
if ((candidate.isDirectory())
|| (fileName.endsWith(completedSuffix))
|| (fileName.startsWith("."))
|| ignorePattern.matcher(fileName).matches()) {
return false;
}
return true;
}
};
List candidateFiles = Arrays.asList(spoolDirectory
.listFiles(filter));
if (candidateFiles.isEmpty()) {
return Optional.absent();
} else {
Collections.sort(candidateFiles, new Comparator() {
public int compare(File a, File b) {
int timeComparison = new Long(a.lastModified())
.compareTo(new Long(b.lastModified()));
if (timeComparison != 0) {
return timeComparison;
} else {
return a.getName().compareTo(b.getName());
}
}
});
File nextFile = candidateFiles.get(0);
try {
// roll the meta file, if needed
String nextPath = nextFile.getPath();
PositionTracker tracker = DurablePositionTracker.getInstance(
metaFile, nextPath);
if (!tracker.getTarget().equals(nextPath)) {
tracker.close();
deleteMetaFile();
tracker = DurablePositionTracker.getInstance(metaFile,
nextPath);
}
// sanity check
Preconditions
.checkState(
tracker.getTarget().equals(nextPath),
"Tracker target %s does not equal expected filename %s",
tracker.getTarget(), nextPath);
ResettableInputStream in = new ResettableFileInputStream(
nextFile, tracker,
ResettableFileInputStream.DEFAULT_BUF_SIZE,
inputCharset, decodeErrorPolicy);
EventDeserializer deserializer = EventDeserializerFactory
.getInstance(deserializerType, deserializerContext, in);
return Optional.of(new FileInfo(nextFile, deserializer));
} catch (FileNotFoundException e) {
// File could have been deleted in the interim
logger.warn("Could not find file: " + nextFile, e);
return Optional.absent();
} catch (IOException e) {
logger.error("Exception opening file: " + nextFile, e);
return Optional.absent();
}
}
}
private void deleteMetaFile() throws IOException {
if (metaFile.exists() && !metaFile.delete()) {
throw new IOException("Unable to delete old meta file " + metaFile);
}
}
/** An immutable class with information about a file being processed. */
private static class FileInfo {
private final File file;
private final long length;
private final long lastModified;
private final EventDeserializer deserializer;
public FileInfo(File file, EventDeserializer deserializer) {
this.file = file;
this.length = file.length();
this.lastModified = file.lastModified();
this.deserializer = deserializer;
}
public long getLength() {
return length;
}
public long getLastModified() {
return lastModified;
}
public EventDeserializer getDeserializer() {
return deserializer;
}
public File getFile() {
return file;
}
}
@InterfaceAudience.Private
@InterfaceStability.Unstable
static enum DeletePolicy {
NEVER, IMMEDIATE, DELAY
}
/**
* Special builder class for ReliableSpoolingFileEventReader
*/
public static class Builder {
private File spoolDirectory;
private String completedSuffix = SpoolDirectorySourceConfigurationExtConstants.SPOOLED_FILE_SUFFIX;
private String ignorePattern = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_IGNORE_PAT;
private String trackerDirPath = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_TRACKER_DIR;
private Boolean annotateFileName = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_FILE_HEADER;
private String fileNameHeader = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_FILENAME_HEADER_KEY;
private Boolean annotateBaseName = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_BASENAME_HEADER;
private String baseNameHeader = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_BASENAME_HEADER_KEY;
private String deserializerType = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_DESERIALIZER;
private Context deserializerContext = new Context();
private String deletePolicy = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_DELETE_POLICY;
private String inputCharset = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_INPUT_CHARSET;
private DecodeErrorPolicy decodeErrorPolicy = DecodeErrorPolicy
.valueOf(SpoolDirectorySourceConfigurationExtConstants.DEFAULT_DECODE_ERROR_POLICY
.toUpperCase());
// 增加代码开始
private Boolean annotateFileNameExtractor = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_FILENAME_EXTRACTOR;
private String fileNameExtractorHeader = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_FILENAME_EXTRACTOR_HEADER_KEY;
private String fileNameExtractorPattern = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_FILENAME_EXTRACTOR_PATTERN;
private Boolean convertToTimestamp = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_FILENAME_EXTRACTOR_CONVERT_TO_TIMESTAMP;
private String dateTimeFormat = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_FILENAME_EXTRACTOR_DATETIME_FORMAT;
private Boolean splitFileName = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_SPLIT_FILENAME;
private String splitBy = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_SPLITY_BY;
private String splitBaseNameHeader = SpoolDirectorySourceConfigurationExtConstants.DEFAULT_SPLIT_BASENAME_HEADER;
public Builder annotateFileNameExtractor(
Boolean annotateFileNameExtractor) {
this.annotateFileNameExtractor = annotateFileNameExtractor;
return this;
}
public Builder fileNameExtractorHeader(String fileNameExtractorHeader) {
this.fileNameExtractorHeader = fileNameExtractorHeader;
return this;
}
public Builder fileNameExtractorPattern(String fileNameExtractorPattern) {
this.fileNameExtractorPattern = fileNameExtractorPattern;
return this;
}
public Builder convertToTimestamp(Boolean convertToTimestamp) {
this.convertToTimestamp = convertToTimestamp;
return this;
}
public Builder dateTimeFormat(String dateTimeFormat) {
this.dateTimeFormat = dateTimeFormat;
return this;
}
public Builder splitFileName(Boolean splitFileName) {
this.splitFileName = splitFileName;
return this;
}
public Builder splitBy(String splitBy) {
this.splitBy = splitBy;
return this;
}
public Builder splitBaseNameHeader(String splitBaseNameHeader) {
this.splitBaseNameHeader = splitBaseNameHeader;
return this;
}
// 增加代码结束
public Builder spoolDirectory(File directory) {
this.spoolDirectory = directory;
return this;
}
public Builder completedSuffix(String completedSuffix) {
this.completedSuffix = completedSuffix;
return this;
}
public Builder ignorePattern(String ignorePattern) {
this.ignorePattern = ignorePattern;
return this;
}
public Builder trackerDirPath(String trackerDirPath) {
this.trackerDirPath = trackerDirPath;
return this;
}
public Builder annotateFileName(Boolean annotateFileName) {
this.annotateFileName = annotateFileName;
return this;
}
public Builder fileNameHeader(String fileNameHeader) {
this.fileNameHeader = fileNameHeader;
return this;
}
public Builder annotateBaseName(Boolean annotateBaseName) {
this.annotateBaseName = annotateBaseName;
return this;
}
public Builder baseNameHeader(String baseNameHeader) {
this.baseNameHeader = baseNameHeader;
return this;
}
public Builder deserializerType(String deserializerType) {
this.deserializerType = deserializerType;
return this;
}
public Builder deserializerContext(Context deserializerContext) {
this.deserializerContext = deserializerContext;
return this;
}
public Builder deletePolicy(String deletePolicy) {
this.deletePolicy = deletePolicy;
return this;
}
public Builder inputCharset(String inputCharset) {
this.inputCharset = inputCharset;
return this;
}
public Builder decodeErrorPolicy(DecodeErrorPolicy decodeErrorPolicy) {
this.decodeErrorPolicy = decodeErrorPolicy;
return this;
}
public ReliableSpoolingFileEventExtReader build() throws IOException {
return new ReliableSpoolingFileEventExtReader(spoolDirectory,
completedSuffix, ignorePattern, trackerDirPath,
annotateFileName, fileNameHeader, annotateBaseName,
baseNameHeader, deserializerType, deserializerContext,
deletePolicy, inputCharset, decodeErrorPolicy,
annotateFileNameExtractor, fileNameExtractorHeader,
fileNameExtractorPattern, convertToTimestamp,
dateTimeFormat, splitFileName, splitBy, splitBaseNameHeader);
}
}
}

复制代码

/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with this
* work for additional information regarding copyright ownership. The ASF
* licenses this file to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations under
* the License.
*/
package com.besttone.flume;
import org.apache.flume.serialization.DecodeErrorPolicy;
public class SpoolDirectorySourceConfigurationExtConstants {
/** Directory where files are deposited. */
public static final String SPOOL_DIRECTORY = "spoolDir";
/** Suffix appended to files when they are finished being sent. */
public static final String SPOOLED_FILE_SUFFIX = "fileSuffix";
public static final String DEFAULT_SPOOLED_FILE_SUFFIX = ".COMPLETED";
/** Header in which to put absolute path filename. */
public static final String FILENAME_HEADER_KEY = "fileHeaderKey";
public static final String DEFAULT_FILENAME_HEADER_KEY = "file";
/** Whether to include absolute path filename in a header. */
public static final String FILENAME_HEADER = "fileHeader";
public static final boolean DEFAULT_FILE_HEADER = false;
/** Header in which to put the basename of file. */
public static final String BASENAME_HEADER_KEY = "basenameHeaderKey";
public static final String DEFAULT_BASENAME_HEADER_KEY = "basename";
/** Whether to include the basename of a file in a header. */
public static final String BASENAME_HEADER = "basenameHeader";
public static final boolean DEFAULT_BASENAME_HEADER = false;
/** What size to batch with before sending to ChannelProcessor. */
public static final String BATCH_SIZE = "batchSize";
public static final int DEFAULT_BATCH_SIZE = 100;
/** Maximum number of lines to buffer between commits. */
@Deprecated
public static final String BUFFER_MAX_LINES = "bufferMaxLines";
@Deprecated
public static final int DEFAULT_BUFFER_MAX_LINES = 100;
/** Maximum length of line (in characters) in buffer between commits. */
@Deprecated
public static final String BUFFER_MAX_LINE_LENGTH = "bufferMaxLineLength";
@Deprecated
public static final int DEFAULT_BUFFER_MAX_LINE_LENGTH = 5000;
/** Pattern of files to ignore */
public static final String IGNORE_PAT = "ignorePattern";
public static final String DEFAULT_IGNORE_PAT = "^[ DISCUZ_CODE_1 ]quot;; // no effect
/** Directory to store metadata about files being processed */
public static final String TRACKER_DIR = "trackerDir";
public static final String DEFAULT_TRACKER_DIR = ".flumespool";
/** Deserializer to use to parse the file data into Flume Events */
public static final String DESERIALIZER = "deserializer";
public static final String DEFAULT_DESERIALIZER = "LINE";
public static final String DELETE_POLICY = "deletePolicy";
public static final String DEFAULT_DELETE_POLICY = "never";
/** Character set used when reading the input. */
public static final String INPUT_CHARSET = "inputCharset";
public static final String DEFAULT_INPUT_CHARSET = "UTF-8";
/** What to do when there is a character set decoding error. */
public static final String DECODE_ERROR_POLICY = "decodeErrorPolicy";
public static final String DEFAULT_DECODE_ERROR_POLICY = DecodeErrorPolicy.FAIL
.name();
public static final String MAX_BACKOFF = "maxBackoff";
public static final Integer DEFAULT_MAX_BACKOFF = 4000;
// 增加代码开始
public static final String FILENAME_EXTRACTOR = "fileExtractor";
public static final boolean DEFAULT_FILENAME_EXTRACTOR = false;
public static final String FILENAME_EXTRACTOR_HEADER_KEY = "fileExtractorHeaderKey";
public static final String DEFAULT_FILENAME_EXTRACTOR_HEADER_KEY = "fileExtractorHeader";
public static final String FILENAME_EXTRACTOR_PATTERN = "fileExtractorPattern";
public static final String DEFAULT_FILENAME_EXTRACTOR_PATTERN = "(.)*";
public static final String FILENAME_EXTRACTOR_CONVERT_TO_TIMESTAMP = "convertToTimestamp";
public static final boolean DEFAULT_FILENAME_EXTRACTOR_CONVERT_TO_TIMESTAMP = false;
public static final String FILENAME_EXTRACTOR_DATETIME_FORMAT = "dateTimeFormat";
public static final String DEFAULT_FILENAME_EXTRACTOR_DATETIME_FORMAT = "yyyy-MM-dd";
public static final String SPLIT_FILENAME = "splitFileName";
public static final boolean DEFAULT_SPLIT_FILENAME = false;
public static final String SPLITY_BY = "splitBy";
public static final String DEFAULT_SPLITY_BY = "\\.";
public static final String SPLIT_BASENAME_HEADER = "splitBaseNameHeader";
public static final String DEFAULT_SPLIT_BASENAME_HEADER = "fileNameSplit";
// 增加代码结束
}

复制代码

package com.besttone.flume;
import static com.besttone.flume.SpoolDirectorySourceConfigurationExtConstants.*;
import java.io.File;
import java.io.IOException;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import org.apache.flume.ChannelException;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDrivenSource;
import org.apache.flume.FlumeException;
import org.apache.flume.conf.Configurable;
import org.apache.flume.instrumentation.SourceCounter;
import org.apache.flume.serialization.DecodeErrorPolicy;
import org.apache.flume.serialization.LineDeserializer;
import org.apache.flume.source.AbstractSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import com.google.common.base.Throwables;
public class SpoolDirectoryExtSource extends AbstractSource implements
Configurable, EventDrivenSource {
private static final Logger logger = LoggerFactory
.getLogger(SpoolDirectoryExtSource.class);
// Delay used when polling for new files
private static final int POLL_DELAY_MS = 500;
/* Config options */
private String completedSuffix;
private String spoolDirectory;
private boolean fileHeader;
private String fileHeaderKey;
private boolean basenameHeader;
private String basenameHeaderKey;
private int batchSize;
private String ignorePattern;
private String trackerDirPath;
private String deserializerType;
private Context deserializerContext;
private String deletePolicy;
private String inputCharset;
private DecodeErrorPolicy decodeErrorPolicy;
private volatile boolean hasFatalError = false;
private SourceCounter sourceCounter;
ReliableSpoolingFileEventExtReader reader;
private ScheduledExecutorService executor;
private boolean backoff = true;
private boolean hitChannelException = false;
private int maxBackoff;
// 增加代码开始
private Boolean annotateFileNameExtractor;
private String fileNameExtractorHeader;
private String fileNameExtractorPattern;
private Boolean convertToTimestamp;
private String dateTimeFormat;
private boolean splitFileName;
private String splitBy;
private String splitBaseNameHeader;
// 增加代码结束
@Override
public synchronized void start() {
logger.info("SpoolDirectorySource source starting with directory: {}",
spoolDirectory);
executor = Executors.newSingleThreadScheduledExecutor();
File directory = new File(spoolDirectory);
try {
reader = new ReliableSpoolingFileEventExtReader.Builder()
.spoolDirectory(directory).completedSuffix(completedSuffix)
.ignorePattern(ignorePattern)
.trackerDirPath(trackerDirPath)
.annotateFileName(fileHeader).fileNameHeader(fileHeaderKey)
.annotateBaseName(basenameHeader)
.baseNameHeader(basenameHeaderKey)
.deserializerType(deserializerType)
.deserializerContext(deserializerContext)
.deletePolicy(deletePolicy).inputCharset(inputCharset)
.decodeErrorPolicy(decodeErrorPolicy)
.annotateFileNameExtractor(annotateFileNameExtractor)
.fileNameExtractorHeader(fileNameExtractorHeader)
.fileNameExtractorPattern(fileNameExtractorPattern)
.convertToTimestamp(convertToTimestamp)
.dateTimeFormat(dateTimeFormat)
.splitFileName(splitFileName).splitBy(splitBy)
.splitBaseNameHeader(splitBaseNameHeader).build();
} catch (IOException ioe) {
throw new FlumeException(
"Error instantiating spooling event parser", ioe);
}
Runnable runner = new SpoolDirectoryRunnable(reader, sourceCounter);
executor.scheduleWithFixedDelay(runner, 0, POLL_DELAY_MS,
TimeUnit.MILLISECONDS);
super.start();
logger.debug("SpoolDirectorySource source started");
sourceCounter.start();
}
@Override
public synchronized void stop() {
executor.shutdown();
try {
executor.awaitTermination(10L, TimeUnit.SECONDS);
} catch (InterruptedException ex) {
logger.info("Interrupted while awaiting termination", ex);
}
executor.shutdownNow();
super.stop();
sourceCounter.stop();
logger.info("SpoolDir source {} stopped. Metrics: {}", getName(),
sourceCounter);
}
@Override
public String toString() {
return "Spool Directory source " + getName() + ": { spoolDir: "
+ spoolDirectory + " }";
}
@Override
public synchronized void configure(Context context) {
spoolDirectory = context.getString(SPOOL_DIRECTORY);
Preconditions.checkState(spoolDirectory != null,
"Configuration must specify a spooling directory");
completedSuffix = context.getString(SPOOLED_FILE_SUFFIX,
DEFAULT_SPOOLED_FILE_SUFFIX);
deletePolicy = context.getString(DELETE_POLICY, DEFAULT_DELETE_POLICY);
fileHeader = context.getBoolean(FILENAME_HEADER, DEFAULT_FILE_HEADER);
fileHeaderKey = context.getString(FILENAME_HEADER_KEY,
DEFAULT_FILENAME_HEADER_KEY);
basenameHeader = context.getBoolean(BASENAME_HEADER,
DEFAULT_BASENAME_HEADER);
basenameHeaderKey = context.getString(BASENAME_HEADER_KEY,
DEFAULT_BASENAME_HEADER_KEY);
batchSize = context.getInteger(BATCH_SIZE, DEFAULT_BATCH_SIZE);
inputCharset = context.getString(INPUT_CHARSET, DEFAULT_INPUT_CHARSET);
decodeErrorPolicy = DecodeErrorPolicy
.valueOf(context.getString(DECODE_ERROR_POLICY,
DEFAULT_DECODE_ERROR_POLICY).toUpperCase());
ignorePattern = context.getString(IGNORE_PAT, DEFAULT_IGNORE_PAT);
trackerDirPath = context.getString(TRACKER_DIR, DEFAULT_TRACKER_DIR);
deserializerType = context
.getString(DESERIALIZER, DEFAULT_DESERIALIZER);
deserializerContext = new Context(context.getSubProperties(DESERIALIZER
+ "."));
// "Hack" to support backwards compatibility with previous generation of
// spooling directory source, which did not support deserializers
Integer bufferMaxLineLength = context
.getInteger(BUFFER_MAX_LINE_LENGTH);
if (bufferMaxLineLength != null && deserializerType != null
&& deserializerType.equalsIgnoreCase(DEFAULT_DESERIALIZER)) {
deserializerContext.put(LineDeserializer.MAXLINE_KEY,
bufferMaxLineLength.toString());
}
maxBackoff = context.getInteger(MAX_BACKOFF, DEFAULT_MAX_BACKOFF);
if (sourceCounter == null) {
sourceCounter = new SourceCounter(getName());
}
//增加代码开始
annotateFileNameExtractor=context.getBoolean(FILENAME_EXTRACTOR, DEFAULT_FILENAME_EXTRACTOR);
fileNameExtractorHeader=context.getString(FILENAME_EXTRACTOR_HEADER_KEY, DEFAULT_FILENAME_EXTRACTOR_HEADER_KEY);
fileNameExtractorPattern=context.getString(FILENAME_EXTRACTOR_PATTERN,DEFAULT_FILENAME_EXTRACTOR_PATTERN);
convertToTimestamp=context.getBoolean(FILENAME_EXTRACTOR_CONVERT_TO_TIMESTAMP, DEFAULT_FILENAME_EXTRACTOR_CONVERT_TO_TIMESTAMP);
dateTimeFormat=context.getString(FILENAME_EXTRACTOR_DATETIME_FORMAT, DEFAULT_FILENAME_EXTRACTOR_DATETIME_FORMAT);
splitFileName=context.getBoolean(SPLIT_FILENAME, DEFAULT_SPLIT_FILENAME);
splitBy=context.getString(SPLITY_BY, DEFAULT_SPLITY_BY);
splitBaseNameHeader=context.getString(SPLIT_BASENAME_HEADER, DEFAULT_SPLIT_BASENAME_HEADER);
//增加代码结束
}
@VisibleForTesting
protected boolean hasFatalError() {
return hasFatalError;
}
/**
* The class always backs off, this exists only so that we can test without
* taking a really long time.
*
* @param backoff
* - whether the source should backoff if the channel is full
*/
@VisibleForTesting
protected void setBackOff(boolean backoff) {
this.backoff = backoff;
}
@VisibleForTesting
protected boolean hitChannelException() {
return hitChannelException;
}
@VisibleForTesting
protected SourceCounter getSourceCounter() {
return sourceCounter;
}
private class SpoolDirectoryRunnable implements Runnable {
private ReliableSpoolingFileEventExtReader reader;
private SourceCounter sourceCounter;
public SpoolDirectoryRunnable(
ReliableSpoolingFileEventExtReader reader,
SourceCounter sourceCounter) {
this.reader = reader;
this.sourceCounter = sourceCounter;
}
@Override
public void run() {
int backoffInterval = 250;
try {
while (!Thread.interrupted()) {
List events = reader.readEvents(batchSize);
if (events.isEmpty()) {
break;
}
sourceCounter.addToEventReceivedCount(events.size());
sourceCounter.incrementAppendBatchReceivedCount();
try {
getChannelProcessor().processEventBatch(events);
reader.commit();
} catch (ChannelException ex) {
logger.warn("The channel is full, and cannot write data now. The "
+ "source will try again after "
+ String.valueOf(backoffInterval)
+ " milliseconds");
hitChannelException = true;
if (backoff) {
TimeUnit.MILLISECONDS.sleep(backoffInterval);
backoffInterval = backoffInterval << 1;
backoffInterval = backoffInterval >= maxBackoff ? maxBackoff
: backoffInterval;
}
continue;
}
backoffInterval = 250;
sourceCounter.addToEventAcceptedCount(events.size());
sourceCounter.incrementAppendBatchAcceptedCount();
}
logger.info("Spooling Directory Source runner has shutdown.");
} catch (Throwable t) {
logger.error(
"FATAL: "
+ SpoolDirectoryExtSource.this.toString()
+ ": "
+ "Uncaught exception in SpoolDirectorySource thread. "
+ "Restart or reconfigure Flume to continue processing.",
t);
hasFatalError = true;
Throwables.propagate(t);
}
}
}
}

复制代码

主要提供了如下几个设置参数：

tier1.sources.source1.type=com.besttone.flume.SpoolDirectoryExtSource #写类的全路径名
tier1.sources.source1.spoolDir=/opt/logs #监控的目录
tier1.sources.source1.splitFileName=true #是否分隔文件名，并把分割后的内容添加到header中，默认false
tier1.sources.source1.splitBy=\\. #以什么符号分隔，默认是"."分隔
tier1.sources.source1.splitBaseNameHeader=fileNameSplit #分割后写入header的key的前缀，比如a.log.2014-07-31,按“."分隔，
则header中有fileNameSplit0=a,fileNameSplit1=log,fileNameSplit2=2014-07-31

（其中还有扩展一个通过正则表达式抽取文件名的功能也在里面，我们这里不用到，就不介绍了）

扩展了这个source之后，前面的那个需求就很容易实现了，只需要：
tier1.sinks.sink1.hdfs.path=hdfs://master68:8020/flume/events/%{fileNameSplit0}/%{fileNameSplit2}
a.log.2014-07-31这个文件的内容就会保存到hdfs://master68:8020/flume/events/a/2014-07-31目录下面去了。

接下来我们说说如何部署这个我们扩展的自定义的spooling directory source（基于CM的设置）。
首先，我们把上面三个类打成JAR包：SpoolDirectoryExtSource.jar
CM的flume插件目录为：/var/lib/flume-ng/plugins.d

然后再你需要使用这个source的agent上的/var/lib/flume-ng/plugins.d目录下面创建SpoolDirectoryExtSource目录以及子目录lib,libext,native,lib是放插件JAR的目录，libext是放插件的依赖JAR的目录，native放使用到的原生库（如果有用到的话）。

我们这里没有使用到其他的依赖，于是就把SpoolDirectoryExtSource.jar放到lib目录下就好了，最终的目录结构：

plugins.d/
plugins.d/SpoolDirectoryExtSource/
plugins.d/SpoolDirectoryExtSource/lib/SpoolDirectoryExtSource.jar
plugins.d/SpoolDirectoryExtSource/libext/
plugins.d/SpoolDirectoryExtSource/native/

复制代码

重新启动flume agent,flume就会自动装载我们的插件，这样在flume.conf中就可以使用全路径类名配置type属性了。

最终flume.conf配置如下：

tier1.sources=source1
tier1.channels=channel1
tier1.sinks=sink1
tier1.sources.source1.type=com.besttone.flume.SpoolDirectoryExtSource
tier1.sources.source1.spoolDir=/opt/logs
tier1.sources.source1.splitFileName=true
tier1.sources.source1.splitBy=\\.
tier1.sources.source1.splitBaseNameHeader=fileNameSplit
tier1.sources.source1.channels=channel1
tier1.sinks.sink1.type=hdfs
tier1.sinks.sink1.channel=channel1
tier1.sinks.sink1.hdfs.path=hdfs://master68:8020/flume/events/%{fileNameSplit0}/%{fileNameSplit2}
tier1.sinks.sink1.hdfs.round=true
tier1.sinks.sink1.hdfs.roundValue=10
tier1.sinks.sink1.hdfs.roundUnit=minute
tier1.sinks.sink1.hdfs.fileType=DataStream
tier1.sinks.sink1.hdfs.writeFormat=Text
tier1.sinks.sink1.hdfs.rollInterval=0
tier1.sinks.sink1.hdfs.rollSize=10240
tier1.sinks.sink1.hdfs.rollCount=0
tier1.sinks.sink1.hdfs.idleTimeout=60
tier1.channels.channel1.type=memory
tier1.channels.channel1.capacity=10000
tier1.channels.channel1.transactionCapacity=1000
tier1.channels.channel1.keep-alive=30

复制代码

附上一张用logger作为sink的查看日志文件的截图：

flume学习（七）、（八）：如何使用event header中的key值以及自定义source_第1张图片

转自：

http://www.aboutyun.com/thread-12043-1-1.html

原文来自：

http://blog.csdn.net/xiao_jun_0820/article/details/38312091

和

http://blog.csdn.net/xiao_jun_0820/article/details/38269079

flume学习（五）：flume将log4j日志数据写入到hdfs
flume学习（六）：使用hive来分析flume收集的日志数据
flume学习（七）、（八）：如何使用event header中的key值以及自定义source
flume学习（九）：自定义拦截器
flume学习（十）：使用Morphline Interceptor
flume学习（十一）：如何使用Spooling Directory Source

你可能感兴趣的:(Flume,转载)

php tofixed,js修复toFixed()四舍五入精度问题，并相对于原生提高了性能 weixin_39590868 php tofixed
思路是将小数分割成整数和小数部分然后进行四舍五入再补全字符串后的0目前没发现有什么问题。有bug请反馈谢谢代码原创，转载请注明连接谢谢"usestrict";/***https://my.oschina.net/isgaoyi/blog/4717508-v1.0(2020-11-15T01:46:32+0800)*[email protected]**修复四舍五入精度问题*@me
前端js变量踩坑，部分手机浏览器不支持let、const weixin_34244102 前端 python 移动开发 ViewUI
2019独角兽企业重金招聘Python工程师标准>>>浏览器通过userAgent判断机型是Android还是iOS，很简单的几行代码，总有iOS用户手机判断不出来。刚开始总以为是判断的错误，优化了好几版，鉴于身边一直没有真机，终是治标不治本，今日偶遇真机，恍然大悟。。。var才是简单js的王道，引以为鉴转载于:https://my.oschina.net/AmosWang/blog/301786
python数字转成字符串 forever1dreamsxx python
转载地址：http://blog.csdn.net/tweller/article/details/7767538数字转成字符串，使用格式化字符串:如tt=322tem='%d'%tttem即为tt转换成的字符串常用的格式化字符串：%d整数%f%F浮点数%e%E科学计数%g%Ge和%f/%E和%F的简写%%输出%格式化操作符辅助指令符号作用*定义宽度或者小数点精度-用做左对齐+在正数前面显示加号(
Burp Suite Professional 2025.1 发布下载，新增功能简介 burp
BurpSuiteProfessional2025.1(macOS,Linux,Windows)-Web应用安全、测试和扫描BurpSuiteProfessional,Test,find,andexploitvulnerabilities.请访问原文链接：https://sysin.org/blog/burp-suite-pro/查看最新版。原创作品，转载请保留出处。作者主页：sysin.orgB
【超详细教程(附源码)】基于 ARM Cortex-M3 处理器与 FPGA 的实时人脸检测 SOC Walker_Lau arm fpga 人脸识别人工智能机器学习
原创作品，转载请联系作者并注明出处：https://github.com/WalkerLau源码地址：https://github.com/WalkerLau/DetectHumanFaces本项目是第四届集成电路创新创业大赛（ARM杯）的参赛作品，包含了详细的技术文档、软件配置教程以及完整的代码。项目描述我们采用ARMCortex-M3软核及FPGA构成了轻量级的实时人脸检测SOC，通过ov56
Cisco ISR 1000 Series IOS XE Release 17.16.1a ED cisco
CiscoISR1000SeriesIOSXERelease17.16.1aED思科1000系列集成多业务路由器IOSXE系统软件请访问原文链接：https://sysin.org/blog/cisco-isr-1000/查看最新版。原创作品，转载请保留出处。作者主页：sysin.org思科1000系列集成多业务路由器可靠性、安全性和性能集互联网接入、高级安全和无线服务功能于一体。思科1000系列
flume+ Elasticsearch +kibana环境搭建及讲解 pincharensheng 大数据 flume kibana elasticsearch 分布式
1、软件介绍1.1、flume1.1.1、flume介绍1）flume概念1、flume是一个分布式的日志收集系统，具有高可靠、高可用、事务管理、失败重启等功能。数据处理速度快，完全可以用于生产环境；2、flume的核心是agent。agent是一个java进程，运行在日志收集端，通过agent接收日志，然后暂存起来，再发送到目的地；3、agent里面包含3个核心组件：source、channel
使用RecyclerView实现电商物流信息的时光轴效果 kirk_wang android 时光轴电商物流
转载请标明出处：http://blog.csdn.net/lmj623565791/article/details/49734867；本文出自:【kirk.wang的博客】相信RecyclerView大家已经不陌生了，最近公司项目中查看物流信息需要用到时光轴，所以在完成此功能之后写下了这一篇文章，希望对大家有所帮助.需求要达到的效果实现功能需求首先为什么要做时光轴，用它来体现有什么好处，现在比较大
科研绘图系列：R语言绘制基因富集GSVA热图（GSVA heatmap）生信学习者1 SCI科研绘图系列 r语言数据可视化数据分析
禁止商业或二改转载，仅供自学使用，侵权必究，如需截取部分内容请后台联系作者!文章目录介绍加载R包数据下载导入数据数据预处理画图1画图2保存图片系统信息介绍科研绘图系列：R语言绘制基因富集GSVA热图（GSVAheatmap）加载R包library(tidyverse)library(dplyr)library(pheatmap)set.seed(123)
python packge index 的使用 weixin_34199335 python 开发工具
pythonpackgeindex（PyPI）是Python编程语言的软件存储库。可用下载各种常用库，以及自己制作并上传库官方网站https://pypi.python.org安装pip3installpymysql安装完的库在python根目录下的lib\sitepackgs\目录下如果需要在命令行窗口直接运行pip3需要配置python根目录下的script目录为环境变量转载于:https:/
我的搬砖工具由 VS Code 变成 Cursor 了老余捞鱼人工智能 AI编程 cursor
作者：老余捞鱼原创不易，转载请标明出处及原作者。写在前面的话：本文介绍了我从VSCode转向Cursor的原因，强调了Cursor的人工智能交互流畅性以及其他一些优于VSCode的特性。VSCode是免费的，而且运行起来非常出色。我一直很喜欢VSCode，不过，它与GitHubCopilot等编码助手扩展的交互并不比使用ChatGPT好多少，而且肯定不如Claude。在寻找其他替代方案时，我偶然发
浏览器沙箱机制 dbp5156 设计模式
简介沙箱模型技术是浏览器和其他应用程序中保护安全的一种组件关系设计模式，最初发明人为GreenBorder公司。2007年5月，谷歌公司收购了该公司，也将此项专利应用于chrome浏览器的研发中。转载于:https://www.cnblogs.com/LO-ME/p/10891824.html
Python的未来 future模块 gftygff
分享一下我老师大神的人工智能教程！零基础，通俗易懂！http://blog.csdn.net/jiangjunshow也欢迎大家转载本篇文章。分享知识，造福人民，实现我们中华民族伟大复兴！转载：http://omencathay.itpub.net/post/30163/414347简单介绍一下python未来将会支持的一些语言特点,虽然Ibm的网站上也有介绍.但是太凌乱了.而且中翻译过后,代码的
11 C++11线程使用 Snow__Sunny #C++11 c++开发语言
声明：该文章转载自此篇文章，欢迎大家支持关注原作者！C++11之前，C++语言没有对并发编程提供语言级别的支持，这使得我们在编写可移植的并发程序时，存在诸多的不便。现在C++11中增加了线程以及线程相关的类，很方便地支持了并发编程，使得编写的多线程程序的可移植性得到了很大的提高。C++11中提供的线程类叫做std::thread，基于这个类创建一个新的线程非常的简单，只需要提供线程函数或者函数对象
spring security配置详解 anzai561156 java
1.IS_AUTHENTICATED_ANONYMOUSLY表示的是匿名用户也可以访问。转载于:https://www.cnblogs.com/Aaronqcd/p/4848517.html
转：Spark RDD算子练习题爱萨萨技术-大数据 spark RDD 练习
版权声明：本文为博主原创文章，遵循CC4.0BY-SA版权协议，转载请附上原文出处链接和本声明。本文链接：https://blog.csdn.net/qq_40825218/article/details/83720732给定数据如下：12张三25男chinese5012张三25男math6012张三25男english7012李四20男chinese5012李四20男math5012李四20男e
Hive数据仓库中的数据导出到MySQL的数据表不成功 sin2201 出错问题数据仓库 hive mysql
可能的原因：(1)没有下载flume和sqoop(2)权限问题：因为MySQL数据库拒绝了root用户从hadoop3主机的连接请求，root用户没有从hadoop3主机进行连接的权限解决：通过MySQL的授权命令来授予权限mysql>GRANTALLPRIVILEGESONsqoop_weblog.*TO'root'@'hadoop3'IDENTIFIEDBY'2020';QueryOK,0ro
高清（200万像素）多灯红外防水枪型网络摄像机 DH-IPC-HFW5200-IRA weixin_30846599 php
DH-IPC-HFW5200-IRA-0722Ahttp://download.dahuatech.com/instruction_download.php?classOne=3907&classTwo=3912&keyword=DH-IPC-HFW5200-IRA-0722Ahttp://www.dahuatech.com/product_detail-449.html转载于:https://w
c# list排序的三种实现方式 CHCH998 c#排序 LIS 实现 list
本文转载自：https://www.cnblogs.com/bradwarden/archive/2012/06/19/2554854.html作者：bradwarden转载请注明该声明。用了一段时间的gridview，对gridview实现的排序功能比较好奇，而且利用C#自带的排序方法只能对某一个字段进行排序，今天demo了一下，总结了三种对list排序的方法，并实现动态传递字段名对list进行
Java力扣题解：169 多数元素——投票法早起之王 leetcode leetcode
题目给定一个大小为n的数组，找到其中的多数元素。多数元素是指在数组中出现次数大于⌊n/2⌋的元素。你可以假设数组是非空的，并且给定的数组总是存在多数元素。来源：力扣（LeetCode）链接：https://leetcode-cn.com/problems/majority-element著作权归领扣网络所有。商业转载请联系官方授权，非商业转载请注明出处。分析这里的投票法，是以第一个元素为基准数，票
idea连接mysql weixin_33758863 开发工具
https://blog.csdn.net/Golden_soft/article/details/80952243转载于:https://www.cnblogs.com/jitipaper/p/10784406.html
一文带你走进车联网安全一泽re. 车联网安全安全网络服务器汽车 iot
免责声明：该⽂章仅供安全学习和技术分享，请勿将该⽂章和⽂章中提到的技术⽤于违法活动上，切勿在⾮授权状态下对其他站点进⾏测试，如产⽣任何后果皆由读者本⼈承担，与作者无关！如有侵权，联系删除，转载请注明出处，感谢！0x00什么是车联网安全车联网安全主要指的是车联网系统的安全，该系统主要分为四个部分，主机、车载T-BOX、手机APP及后台系统。主机主要用于车内的影音娱乐，以及车辆信息显示也就是IVI或者
Windows Server 2022 中文版、英文版下载 (2024 年 12 月更新) windows-server
WindowsServer2022中文版、英文版下载(2024年12月更新)WindowsServer2022x64,Version21H2(updatedDec2024)请访问原文链接：https://sysin.org/blog/windows-server-2022/查看最新版。原创作品，转载请保留出处。作者主页：sysin.orgWindowsServer2022采用先进的多层安全机制、A
Windows Server 2025 中文版、英文版下载 (2024 年 11 月更新) windows-server
WindowsServer2025中文版、英文版下载(2024年11月更新)WindowsServer2025LTSCx64(updatedNov2024)请访问原文链接：https://sysin.org/blog/windows-server-2025/查看最新版。原创作品，转载请保留出处。作者主页：sysin.org2024年11月3日，微软宣布WindowsServer2025从Build
【软件开发 | 甲方建设】业务系统有多复杂？秋说前后端项目开发(新手必知必会)业务系统软件开发重构敏捷开发
本文转载自腾讯云官方社区公众号，对原文内容做二次修改。文章目录01前言02斜体样式系统复杂性软件复用性和开发成本03软件开发提效业务代码杂乱04业务系统复杂的根本原因不可避免的代码腐化敏捷开发模式中华田园式敏捷FreeasBeervsFreeasPuppy05写在最后01前言过去一整年我都一直在思考「业务系统复杂性」这个问题。其实对这个问题，从我开始工作就有不断地思考，不过这些思考大多在读完一两本
Double值保留两位小数的四种方法 Small_Mayi java
Double值保留两位小数的四种方法转载https://blog.csdn.net/hello_word2/article/details/81808827
python消费kafka数据nginx日志实时_Openresty+Lua+Kafka实现日志实时采集 weixin_39997311
简介在很多数据采集场景下，Flume作为一个高性能采集日志的工具，相信大家都知道它。许多人想起Flume这个组件能联想到的大多数都是Flume跟Kafka相结合进行日志的采集，这种方案有很多他的优点，比如高性能、高吞吐、数据可靠性等。但是我们如果要求对日志进行实时的采集，这显然不是一个好的解决方案。原因如下：就目前来说，Flume能支持实时监控一个目录的数据文件，一旦对某个目录的文件采集完成，就会
openresty+lua实现实时写kafka sky@梦幻未来大数据 openresty nginx openresty lua
一.背景在使用openresty+lua+nginx+flume，通过定时切分日志发送kafka的方式无法满足实时性的情况下，小编开始研究openresty+lua+nginx+kafka实时写kafka，从而达到数据实时性，和高性能保证。二.实现1.openresty安装nginx，以及lua的使用请看博主上一篇博客https://blog.csdn.net/qq_29497387/articl
spss因子分析过程中，旋转载荷平方和累积有点低咋办怎么调整 xiamu_CDA python
SPSS因子分析过程中，旋转载荷平方和累积有点低咋办？怎么调整？在数据分析领域，因子分析是一项重要的统计技术，尤其在心理学、社会学、市场营销等领域中应用广泛。它通过将多个变量简化为少数几个潜在因子，帮助研究者理解变量之间的内在结构。然而，在实际操作过程中，我们常常会遇到一些棘手的问题，比如旋转载荷平方和累积值偏低。这不仅会影响模型的解释力，还可能导致研究结果的可靠性大打折扣。那么，当我们在使用SP
Nexpose 7.3.2 for Linux & Windows - 漏洞扫描漏洞
Nexpose7.3.2forLinux&Windows-漏洞扫描Rapid7on-premVulnerabilityManagement,releasedJan23,2025请访问原文链接：https://sysin.org/blog/nexpose-7/查看最新版。原创作品，转载请保留出处。作者主页：sysin.orgNexposeVulnerabilityScanner您的本地漏洞扫描程序搜
html 周华华 html
js 1，数组的排列 var arr=[1,4,234,43,52,]; for(var x=0;x<arr.length;x++){ for(var y=x-1;y<arr.length;y++){ if(arr[x]<arr[y]){ &
【Struts2 四】Struts2拦截器 bit1129 struts2拦截器
Struts2框架是基于拦截器实现的，可以对某个Action进行拦截，然后某些逻辑处理，拦截器相当于AOP里面的环绕通知，即在Action方法的执行之前和之后根据需要添加相应的逻辑。事实上，即使struts.xml没有任何关于拦截器的配置，Struts2也会为我们添加一组默认的拦截器，最常见的是，请求参数自动绑定到Action对应的字段上。 Struts2中自定义拦截器的步骤是：
make:cc 命令未找到解决方法 daizj linux 命令未知 make cc
安装rz sz程序时，报下面错误： [root@slave2 src]# make posix cc -O -DPOSIX -DMD=2 rz.c -o rz make: cc：命令未找到 make: *** [posix] 错误 127 系统：centos 6.6 环境：虚拟机错误原因：系统未安装gcc，这个是由于在安
Oracle之Job应用周凡杨 oracle job
最近写服务，服务上线后，需要写一个定时执行的SQL脚本，清理并更新数据库表里的数据，应用到了Oracle 的 Job的相关知识。在此总结一下。一：查看相关job信息 1、相关视图 dba_jobs all_jobs user_jobs dba_jobs_running 包含正在运行
多线程机制朱辉辉33 多线程
转至http://blog.csdn.net/lj70024/archive/2010/04/06/5455790.aspx 程序、进程和线程：程序是一段静态的代码，它是应用程序执行的蓝本。进程是程序的一次动态执行过程，它对应了从代码加载、执行至执行完毕的一个完整过程，这个过程也是进程本身从产生、发展至消亡的过程。线程是比进程更小的单位，一个进程执行过程中可以产生多个线程，每个线程有自身的
web报表工具FineReport使用中遇到的常见报错及解决办法（一）老A不折腾 web报表 finereport java报表报表工具
FineReport使用中遇到的常见报错及解决办法（一）这里写点抛砖引玉，希望大家能把自己整理的问题及解决方法晾出来，Mark一下，利人利己。出现问题先搜一下文档上有没有，再看看度娘有没有，再看看论坛有没有。有报错要看日志。下面简单罗列下常见的问题，大多文档上都有提到的。 1、address pool is full：含义：地址池满，连接数超过并发数上
mysql rpm安装后没有my.cnf 林鹤霄没有my.cnf
Linux下用rpm包安装的MySQL是不会安装/etc/my.cnf文件的，至于为什么没有这个文件而MySQL却也能正常启动和作用，在这儿有两个说法，第一种说法，my.cnf只是MySQL启动时的一个参数文件，可以没有它，这时MySQL会用内置的默认参数启动，第二种说法，MySQL在启动时自动使用/usr/share/mysql目录下的my-medium.cnf文件，这种说法仅限于r
Kindle Fire HDX root并安装谷歌服务框架之后仍无法登陆谷歌账号的问题 aigo root
原文：http://kindlefireforkid.com/how-to-setup-a-google-account-on-amazon-fire-tablet/ Step 4: Run ADB command from your PC On the PC, you need install Amazon Fire ADB driver and instal
javascript 中var提升的典型实例 alxw4616 JavaScript
// 刚刚在书上看到的一个小问题,很有意思.大家一起思考下吧 myname = 'global'; var fn = function () { console.log(myname); // undefined var myname = 'local'; console.log(myname); // local }; fn() // 上述代码实际上等同于以下代码 m
定时器和获取时间的使用百合不是茶时间的转换定时器
定时器:定时创建任务在游戏设计的时候用的比较多 Timer();定时器 TImerTask();Timer的子类由 Timer 安排为一次执行或重复执行的任务。定时器类Timer在java.util包中。使用时，先实例化，然后使用实例的schedule(TimerTask task, long delay)方法，设定
JDK1.5 Queue bijian1013 java thread java多线程 Queue
JDK1.5 Queue LinkedList： LinkedList不是同步的。如果多个线程同时访问列表，而其中至少一个线程从结构上修改了该列表，则它必须保持外部同步。（结构修改指添加或删除一个或多个元素的任何操作；仅设置元素的值不是结构修改。）这一般通过对自然封装该列表的对象进行同步操作来完成。如果不存在这样的对象，则应该使用 Collections.synchronizedList 方
http认证原理和https bijian1013 http https
一.基础介绍在URL前加https://前缀表明是用SSL加密的。你的电脑与服务器之间收发的信息传输将更加安全。 Web服务器启用SSL需要获得一个服务器证书并将该证书与要使用SSL的服务器绑定。 http和https使用的是完全不同的连接方式，用的端口也不一样,前者是80，后
【Java范型五】范型继承 bit1129 java
定义如下一个抽象的范型类，其中定义了两个范型参数，T1，T2 package com.tom.lang.generics; public abstract class SuperGenerics<T1, T2> { private T1 t1; private T2 t2; public abstract void doIt(T
【Nginx六】nginx.conf常用指令(Directive) bit1129 Directive
1. worker_processes 8; 表示Nginx将启动8个工作者进程，通过ps -ef|grep nginx,会发现有8个Nginx Worker Process在运行 nobody 53879 118449 0 Apr22 ? 00:26:15 nginx: worker process
lua 遍历Header头部 ronin47 lua header 遍历　
local headers = ngx.req.get_headers() ngx.say("headers begin", "<br/>") ngx.say("Host : ", he
java-32.通过交换a,b中的元素，使[序列a元素的和]与[序列b元素的和]之间的差最小(两数组的差最小)。 bylijinnan java
import java.util.Arrays; public class MinSumASumB { /** * Q32.有两个序列a,b，大小都为n,序列元素的值任意整数，无序. * * 要求：通过交换a,b中的元素，使[序列a元素的和]与[序列b元素的和]之间的差最小。 * 例如: * int[] a = {100,99,98,1,2,3
redis 开窍的石头 redis
在redis的redis.conf配置文件中找到# requirepass foobared 把它替换成requirepass 12356789 后边的12356789就是你的密码打开redis客户端输入config get requirepass 返回 redis 127.0.0.1:6379> config get requirepass 1) "require
[JAVA图像与图形]现有的GPU架构支持JAVA语言吗？ comsci java语言
无论是opengl还是cuda，都是建立在C语言体系架构基础上的，在未来，图像图形处理业务快速发展，相关领域市场不断扩大的情况下，我们JAVA语言系统怎么从这么庞大，且还在不断扩大的市场上分到一块蛋糕，是值得每个JAVAER认真思考和行动的事情
安装ubuntu14.04登录后花屏了怎么办 cuiyadll ubuntu
这个情况，一般属于显卡驱动问题。可以先尝试安装显卡的官方闭源驱动。按键盘三个键：CTRL + ALT + F1 进入终端，输入用户名和密码登录终端：安装amd的显卡驱动 sudo apt-get install fglrx 安装nvidia显卡驱动 sudo ap
SSL 与数字证书的基本概念和工作原理 darrenzhu 加密 ssl 证书密钥签名
SSL 与数字证书的基本概念和工作原理 http://www.linuxde.net/2012/03/8301.html SSL握手协议的目的是或最终结果是让客户端和服务器拥有一个共同的密钥，握手协议本身是基于非对称加密机制的，之后就使用共同的密钥基于对称加密机制进行信息交换。 http://www.ibm.com/developerworks/cn/webspher
Ubuntu设置ip的步骤 dcj3sjt126com ubuntu
在单位的一台机器完全装了Ubuntu Server，但回家只能在XP上VM一个，装的时候网卡是DHCP的，用ifconfig查了一下ip是192.168.92.128,可以ping通。转载不是错： Ubuntu命令行修改网络配置方法 /etc/network/interfaces打开后里面可设置DHCP或手动设置静态ip。前面auto eth0，让网卡开机自动挂载. 1. 以D
php包管理工具推荐 dcj3sjt126com PHP Composer
http://www.phpcomposer.com/ Composer是 PHP 用来管理依赖（dependency）关系的工具。你可以在自己的项目中声明所依赖的外部工具库（libraries），Composer 会帮你安装这些依赖的库文件。中文文档入门指南下载安装包列表 Composer 中国镜像
Gson使用四（TypeAdapter） eksliang json gson Gson自定义转换器 gsonTypeAdapter
转载请出自出处：http://eksliang.iteye.com/blog/2175595 一.概述 Gson的TypeAapter可以理解成自定义序列化和返序列化二、应用场景举例例如我们通常去注册时（那些外国网站），会让我们输入firstName，lastName,但是转到我们都
JQM控件之Navbar和Tabs gundumw100 html xml css
在JQM中使用导航栏Navbar是简单的。只需要将data-role="navbar"赋给div即可： <div data-role="navbar"> <ul> <li><a href="#" class="ui-btn-active&qu
利用归并排序算法对大文件进行排序 iwindyforest java 归并排序大文件分治法 Merge sort
归并排序算法介绍，请参照Wikipeida zh.wikipedia.org/wiki/%E5%BD%92%E5%B9%B6%E6%8E%92%E5%BA%8F 基本思想：大文件分割成行数相等的两个子文件，递归（归并排序）两个子文件，直到递归到分割成的子文件低于限制行数低于限制行数的子文件直接排序两个排序好的子文件归并到父文件直到最后所有排序好的父文件归并到输入
iOS UIWebView URL拦截啸笑天 UIWebView
本文译者：candeladiao，原文：URL filtering for UIWebView on the iPhone说明：译者在做app开发时，因为页面的javascript文件比较大导致加载速度很慢，所以想把javascript文件打包在app里，当UIWebView需要加载该脚本时就从app本地读取，但UIWebView并不支持加载本地资源。最后从下文中找到了解决方法，第一次翻译，难免有
索引的碎片整理SQL语句 macroli sql
SET NOCOUNT ON DECLARE @tablename VARCHAR (128) DECLARE @execstr VARCHAR (255) DECLARE @objectid INT DECLARE @indexid INT DECLARE @frag DECIMAL DECLARE @maxfrag DECIMAL --设置最大允许的碎片数量,超过则对索引进行碎片
Angularjs同步操作http请求with $promise qiaolevip 每天进步一点点学习永无止境 AngularJS 纵观千象
// Define a factory app.factory('profilePromise', ['$q', 'AccountService', function($q, AccountService) { var deferred = $q.defer(); AccountService.getProfile().then(function(res) {
hibernate联合查询问题 sxj19881213 sql Hibernate HQL 联合查询
最近在用hibernate做项目，遇到了联合查询的问题，以及联合查询中的N+1问题。针对无外键关联的联合查询，我做了HQL和SQL的实验，希望能帮助到大家。（我使用的版本是hibernate3.3.2） 1 几个常识：（1）hql中的几种join查询，只有在外键关联、并且作了相应配置时才能使用。（2）hql的默认查询策略，在进行联合查询时，会产
struts2.xml wuai struts
<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE struts PUBLIC "-//Apache Software Foundation//DTD Struts Configuration 2.3//EN" "http://struts.apache