canal 源码解析（1）-启动篇（3）

1.0）meta 为一个接口，有多种实现方式，

if (!metaManager.isStart()) {
    metaManager.start();
}

因为配置文件已经指定了实现模式，所以进入filemixedmetamanager模式

先看一下整个start方法

public void start() {
    super.start();
    Assert.notNull(dataDir);
    if (!dataDir.exists()) {
        try {
            FileUtils.forceMkdir(dataDir);
        } catch (IOException e) {
            throw new CanalMetaManagerException(e);
        }
    }

    if (!dataDir.canRead() || !dataDir.canWrite()) {
        throw new CanalMetaManagerException("dir[" + dataDir.getPath() + "] can not read/write");
    }

    dataFileCaches = MigrateMap.makeComputingMap(new Function, File>() {

        public File apply(String destination) {
            return getDataFile(destination);
        }
    });

    executor = Executors.newScheduledThreadPool(1);
    destinations = MigrateMap.makeComputingMap(new Function, List>() {

        public List apply(String destination) {
            return loadClientIdentity(destination);
        }
    });

    cursors = MigrateMap.makeComputingMap(new Function, Position>() {

        public Position apply(ClientIdentity clientIdentity) {
            Position position = loadCursor(clientIdentity.getDestination(), clientIdentity);
            if (position == null) {
                return nullCursor; // 返回一个空对象标识，避免出现异常
            } else {
                return position;
            }
        }
    });

    updateCursorTasks = Collections.synchronizedSet(new HashSet());

    // 启动定时工作任务
    executor.scheduleAtFixedRate(new Runnable() {

        public void run() {
            List tasks = new ArrayList(updateCursorTasks);
            for (ClientIdentity clientIdentity : tasks) {
                MDC.put("destination", String.valueOf(clientIdentity.getDestination()));
                try {
                    // 定时将内存中的最新值刷到file中，多次变更只刷一次
                    if (logger.isInfoEnabled()) {
                        LogPosition cursor = (LogPosition) getCursor(clientIdentity);
                        logger.info("clientId:{} cursor:[{},{},{},{},{}] address[{}]", new Object[] {
                                clientIdentity.getClientId(), cursor.getPostion().getJournalName(),
                                cursor.getPostion().getPosition(), cursor.getPostion().getTimestamp(),
                                cursor.getPostion().getServerId(), cursor.getPostion().getGtid(),
                                cursor.getIdentity().getSourceAddress().toString() });
                    }
                    flushDataToFile(clientIdentity.getDestination());
                    updateCursorTasks.remove(clientIdentity);
                } catch (Throwable e) {
                    // ignore
                    logger.error("period update" + clientIdentity.toString() + " curosr failed!", e);
                }
            }
        }
    },
        period,
        period,
        TimeUnit.MILLISECONDS);
}

 文件是否存在，是否可读可写，初始化类，启动定时任务，一秒后执行，一秒执行一次

定时任务里，对文件的处理

private void flushDataToFile(String destination) {
    flushDataToFile(destination, dataFileCaches.get(destination));
}

private void flushDataToFile(String destination, File dataFile) {
    FileMetaInstanceData data = new FileMetaInstanceData();
    if (destinations.containsKey(destination)) {
        synchronized (destination.intern()) { // 基于destination控制一下并发更新
            data.setDestination(destination);

            List clientDatas = Lists.newArrayList();
            List clientIdentitys = destinations.get(destination);
            for (ClientIdentity clientIdentity : clientIdentitys) {
                FileMetaClientIdentityData clientData = new FileMetaClientIdentityData();
                clientData.setClientIdentity(clientIdentity);
                Position position = cursors.get(clientIdentity);
                if (position != null && position != nullCursor) {
                    clientData.setCursor((LogPosition) position);
                }

                clientDatas.add(clientData);
            }

            data.setClientDatas(clientDatas);
        }

        String json = JsonUtils.marshalToString(data);
        try {
            FileUtils.writeStringToFile(dataFile, json);
        } catch (IOException e) {
            throw new CanalMetaManagerException(e);
        }
    }
}

* 1. 先写内存，然后定时刷新数据到File
* 2. 数据采取overwrite模式(只保留最后一次)，通过logger实施append模式(记录历史版本)

文件meta.dat里存放的是

client 已成功消费的最后binlog位点，时间，实例 的数据。目录和canal同级别，可配置，在canal.instance里配置

canal.conf.dir = ../conf

meta.dat里的文件如下：

{"clientDatas":[{"clientIdentity":{"clientId":1001,"destination":"example","filter":""},"cursor":{"identity":{"slaveId":-1,"sourceAddress":{"address":"DESKTOP-B1R6VMO","port":3306}},"postion":{"gtid":"","included":false,"journalName":"mysql-bin.000009","position":6218,"serverId":1,"timestamp":1527665906000}}}],"destination":"example"}

1.1）alarm没有做任何处理，就log打印了一下

if (!alarmHandler.isStart()) {
    alarmHandler.start();
}

1.2）store 初始化内存  new了一个数组 size为16384-1

if (!eventStore.isStart()) {
    eventStore.start();
}

public void start() throws CanalStoreException {
    super.start();
    if (Integer.bitCount(bufferSize) != 1) {
        throw new IllegalArgumentException("bufferSize must be a power of 2");
    }

    indexMask = bufferSize - 1;
    entries = new Event[bufferSize];
}

其中数组中的canalenty为google的序列化protrobuf

1.3）sink 

if (!eventSink.isStart()) {
    eventSink.start();
}

public void start() {
    super.start();
    Assert.notNull(eventStore);

    for (CanalEventDownStreamHandler handler : getHandlers()) {
        if (!handler.isStart()) {
            handler.start();
        }
    }
}

1.4）parse

if (!eventParser.isStart()) {
    beforeStartEventParser(eventParser);
    eventParser.start();
    afterStartEventParser(eventParser);
}

before 做解析前处理，启动logposition和hacontroller

protected void startEventParserInternal(CanalEventParser eventParser, boolean isGroup) {
    if (eventParser instanceof AbstractEventParser) {
        AbstractEventParser abstractEventParser = (AbstractEventParser) eventParser;
        // 首先启动log position管理器
        CanalLogPositionManager logPositionManager = abstractEventParser.getLogPositionManager();
        if (!logPositionManager.isStart()) {
            logPositionManager.start();
        }
    }

    if (eventParser instanceof MysqlEventParser) {
        MysqlEventParser mysqlEventParser = (MysqlEventParser) eventParser;
        CanalHAController haController = mysqlEventParser.getHaController();

        if (haController instanceof HeartBeatHAController) {
            ((HeartBeatHAController) haController).setCanalHASwitchable(mysqlEventParser);
        }

        if (!haController.isStart()) {
            haController.start();
        }

    }
}

1.4.2) 比较复杂的代码来了，和主库连接，发送dump指令，解析等

public void start() throws CanalParseException {
    if (runningInfo == null) { // 第一次链接主库
        runningInfo = masterInfo;
    }

    super.start();
}

public void start() throws CanalParseException {
    if (enableTsdb) {
        if (tableMetaTSDB == null) {
            // 初始化
            tableMetaTSDB = TableMetaTSDBBuilder.build(destination, tsdbSpringXml);
        }
    }

    super.start();
}

初始化缓存队列，线程，全局变量循环发送命令。

public void start() {
    super.start();
    MDC.put("destination", destination);
    // 配置transaction buffer
    // 初始化缓冲队列
    transactionBuffer.setBufferSize(transactionSize);// 设置buffer大小
    transactionBuffer.start();
    // 构造bin log parser
    binlogParser = buildParser();// 初始化一下BinLogParser
    binlogParser.start();
    // 启动工作线程
    parseThread = new Thread(new Runnable() {

        public void run() {
            MDC.put("destination", String.valueOf(destination));
            ErosaConnection erosaConnection = null;
            while (running) {
                try {
                    // 开始执行replication
                    // 1. 构造Erosa连接
                    erosaConnection = buildErosaConnection();

                    // 2. 启动一个心跳线程
                    startHeartBeat(erosaConnection);

                    // 3. 执行dump前的准备工作
                    preDump(erosaConnection);

                    erosaConnection.connect();// 链接
                    // 4. 获取最后的位置信息
                    EntryPosition position = findStartPosition(erosaConnection);
                    final EntryPosition startPosition = position;
                    if (startPosition == null) {
                        throw new CanalParseException("can't find start position for " + destination);
                    }

                    if (!processTableMeta(startPosition)) {
                        throw new CanalParseException("can't find init table meta for " + destination
                                                      + " with position : " + startPosition);
                    }
                    logger.warn("find start position : {}", startPosition.toString());
                    // 重新链接，因为在找position过程中可能有状态，需要断开后重建
                    erosaConnection.reconnect();

                    final SinkFunction sinkHandler = new SinkFunction<EVENT>() {

                        private LogPosition lastPosition;

                        public boolean sink(EVENT event) {
                            try {
                                CanalEntry.Entry entry = parseAndProfilingIfNecessary(event, false);

                                if (!running) {
                                    return false;
                                }

                                if (entry != null) {
                                    exception = null; // 有正常数据流过，清空exception
                                    transactionBuffer.add(entry);
                                    // 记录一下对应的positions
                                    this.lastPosition = buildLastPosition(entry);
                                    // 记录一下最后一次有数据的时间
                                    lastEntryTime = System.currentTimeMillis();
                                }
                                return running;
                            } catch (TableIdNotFoundException e) {
                                throw e;
                            } catch (Throwable e) {
                                if (e.getCause() instanceof TableIdNotFoundException) {
                                    throw (TableIdNotFoundException) e.getCause();
                                }
                                // 记录一下，出错的位点信息
                                processSinkError(e,
                                    this.lastPosition,
                                    startPosition.getJournalName(),
                                    startPosition.getPosition());
                                throw new CanalParseException(e); // 继续抛出异常，让上层统一感知
                            }
                        }

                    };

                    // 4. 开始dump数据
                    // 判断所属instance是否启用GTID模式，是的话调用ErosaConnection中GTID对应方法dump数据
                    if (isGTIDMode()) {
                        erosaConnection.dump(MysqlGTIDSet.parse(startPosition.getGtid()), sinkHandler);
                    } else {
                        if (StringUtils.isEmpty(startPosition.getJournalName()) && startPosition.getTimestamp() != null) {
                            erosaConnection.dump(startPosition.getTimestamp(), sinkHandler);
                        } else {
                            erosaConnection.dump(startPosition.getJournalName(),
                                    startPosition.getPosition(),
                                    sinkHandler);
                        }
                    }

                } catch (TableIdNotFoundException e) {
                    exception = e;
                    // 特殊处理TableIdNotFound异常,出现这样的异常，一种可能就是起始的position是一个事务当中，导致tablemap
                    // Event时间没解析过
                    needTransactionPosition.compareAndSet(false, true);
                    logger.error(String.format("dump address %s has an error, retrying. caused by ",
                        runningInfo.getAddress().toString()), e);
                } catch (Throwable e) {
                    processDumpError(e);
                    exception = e;
                    if (!running) {
                        if (!(e instanceof java.nio.channels.ClosedByInterruptException || e.getCause() instanceof java.nio.channels.ClosedByInterruptException)) {
                            throw new CanalParseException(String.format("dump address %s has an error, retrying. ",
                                runningInfo.getAddress().toString()), e);
                        }
                    } else {
                        logger.error(String.format("dump address %s has an error, retrying. caused by ",
                            runningInfo.getAddress().toString()), e);
                        sendAlarm(destination, ExceptionUtils.getFullStackTrace(e));
                    }
                } finally {
                    // 重新置为中断状态
                    Thread.interrupted();
                    // 关闭一下链接
                    afterDump(erosaConnection);
                    try {
                        if (erosaConnection != null) {
                            erosaConnection.disconnect();
                        }
                    } catch (IOException e1) {
                        if (!running) {
                            throw new CanalParseException(String.format("disconnect address %s has an error, retrying. ",
                                runningInfo.getAddress().toString()),
                                e1);
                        } else {
                            logger.error("disconnect address {} has an error, retrying., caused by ",
                                runningInfo.getAddress().toString(),
                                e1);
                        }
                    }
                }
                // 出异常了，退出sink消费，释放一下状态
                eventSink.interrupt();
                transactionBuffer.reset();// 重置一下缓冲队列，重新记录数据
                binlogParser.reset();// 重新置位

                if (running) {
                    // sleep一段时间再进行重试
                    try {
                        Thread.sleep(10000 + RandomUtils.nextInt(10000));
                    } catch (InterruptedException e) {
                    }
                }
            }
            MDC.remove("destination");
        }
    });

    parseThread.setUncaughtExceptionHandler(handler);
    parseThread.setName(String.format("destination = %s , address = %s , EventParser",
        destination,
        runningInfo == null ? null : runningInfo.getAddress()));
    parseThread.start();
}

解析循环里的5步操作

 构造一个connection 

// 1. 构造Erosa连接
erosaConnection = buildErosaConnection();

private MysqlConnection buildMysqlConnection(AuthenticationInfo runningInfo) {
    MysqlConnection connection = new MysqlConnection(runningInfo.getAddress(),
        runningInfo.getUsername(),
        runningInfo.getPassword(),
        connectionCharsetNumber,
        runningInfo.getDefaultDatabaseName());
    connection.getConnector().setReceiveBufferSize(receiveBufferSize);
    connection.getConnector().setSendBufferSize(sendBufferSize);
    connection.getConnector().setSoTimeout(defaultConnectionTimeoutInSeconds * 1000);
    connection.setCharset(connectionCharset);
    // 随机生成slaveId
    if (this.slaveId <= 0) {
        this.slaveId = generateUniqueServerId();
    }
    connection.setSlaveId(this.slaveId);
    return connection;
}

返回一个mysqlconnection实体bean（user，password，url scheme），并把验证信息set里

2、启动一个线程做心跳（默认3秒 把type设置为hearbeat放到enty里 ，采用timer定时任务，）

// 2. 启动一个心跳线程
startHeartBeat(erosaConnection);

protected void startHeartBeat(ErosaConnection connection) {
    lastEntryTime = 0L; // 初始化
    if (timer == null) {// lazy初始化一下
        String name = String.format("destination = %s , address = %s , HeartBeatTimeTask",
            destination,
            runningInfo == null ? null : runningInfo.getAddress().toString());
        synchronized (AbstractEventParser.class) {
            // synchronized (MysqlEventParser.class) {
            // why use MysqlEventParser.class, u know, MysqlEventParser is
            // the child class 4 AbstractEventParser,
            // do this is ...
            if (timer == null) {
                timer = new Timer(name, true);
            }
        }
    }

    if (heartBeatTimerTask == null) {// fixed issue #56，避免重复创建heartbeat线程
        heartBeatTimerTask = buildHeartBeatTimeTask(connection);
        Integer interval = detectingIntervalInSeconds;
        timer.schedule(heartBeatTimerTask, interval * 1000L, interval * 1000L);
        logger.info("start heart beat.... ");
    }
}

包装entry ，发送到sink ，其实里面为心跳数据

protected TimerTask buildHeartBeatTimeTask(ErosaConnection connection) {
    return new TimerTask() {

        public void run() {
            try {
                if (exception == null || lastEntryTime > 0) {
                    // 如果未出现异常，或者有第一条正常数据
                    long now = System.currentTimeMillis();
                    long inteval = (now - lastEntryTime) / 1000;
                    if (inteval >= detectingIntervalInSeconds) {
                        Header.Builder headerBuilder = Header.newBuilder();
                        headerBuilder.setExecuteTime(now);
                        Entry.Builder entryBuilder = Entry.newBuilder();
                        entryBuilder.setHeader(headerBuilder.build());
                        entryBuilder.setEntryType(EntryType.HEARTBEAT);
                        Entry entry = entryBuilder.build();
                        // 提交到sink中，目前不会提交到store中，会在sink中进行忽略
                        consumeTheEventAndProfilingIfNecessary(Arrays.asList(entry));
                    }
                }

            } catch (Throwable e) {
                logger.warn("heartBeat run failed ", e);
            }
        }

    };
}

发送到sink

protected boolean consumeTheEventAndProfilingIfNecessary(List entrys) throws CanalSinkException,
                                                                                       InterruptedException {
    long startTs = -1;
    boolean enabled = getProfilingEnabled();
    if (enabled) {
        startTs = System.currentTimeMillis();
    }

    boolean result = eventSink.sink(entrys, (runningInfo == null) ? null : runningInfo.getAddress(), destination);

    if (enabled) {
        this.processingInterval = System.currentTimeMillis() - startTs;
    }

    if (consumedEventCount.incrementAndGet() < 0) {
        consumedEventCount.set(0);
    }

    return result;
}

3、执行前dump工作 数据库连接是否正常

// 3. 执行dump前的准备工作
preDump(erosaConnection);

protected void preDump(ErosaConnection connection) {
    if (!(connection instanceof MysqlConnection)) {
        throw new CanalParseException("Unsupported connection type : " + connection.getClass().getSimpleName());
    }

    if (binlogParser != null && binlogParser instanceof LogEventConvert) {
        metaConnection = (MysqlConnection) connection.fork();
        try {
            metaConnection.connect();
        } catch (IOException e) {
            throw new CanalParseException(e);
        }

        if (supportBinlogFormats != null && supportBinlogFormats.length > 0) {
            BinlogFormat format = ((MysqlConnection) metaConnection).getBinlogFormat();
            boolean found = false;
            for (BinlogFormat supportFormat : supportBinlogFormats) {
                if (supportFormat != null && format == supportFormat) {
                    found = true;
                    break;
                }
            }

            if (!found) {
                throw new CanalParseException("Unsupported BinlogFormat " + format);
            }
        }

        if (supportBinlogImages != null && supportBinlogImages.length > 0) {
            BinlogImage image = ((MysqlConnection) metaConnection).getBinlogImage();
            boolean found = false;
            for (BinlogImage supportImage : supportBinlogImages) {
                if (supportImage != null && image == supportImage) {
                    found = true;
                    break;
                }
            }

            if (!found) {
                throw new CanalParseException("Unsupported BinlogImage " + image);
            }
        }

        if (tableMetaTSDB != null && tableMetaTSDB instanceof DatabaseTableMeta) {
            ((DatabaseTableMeta) tableMetaTSDB).setConnection(metaConnection);
            ((DatabaseTableMeta) tableMetaTSDB).setFilter(eventFilter);
            ((DatabaseTableMeta) tableMetaTSDB).setBlackFilter(eventBlackFilter);
        }

        tableMetaCache = new TableMetaCache(metaConnection, tableMetaTSDB);
        ((LogEventConvert) binlogParser).setTableMetaCache(tableMetaCache);
    }
}

判断数据库binlog模式和同步类型

binlog格式进行过滤，默认ROW模式。 

 binlog image进行过滤，默认是FULL，变更前后的数据，为minimal，变更后的值。 

构造表结构源数据的缓存TableMetaCache

4、获取binlog位置信息 ，启动时候，只做初始化，然后就结束了（当数据流过才会有，加载配置，监听等）现在来解析一下findstartposition（）方法

// 4. 获取最后的位置信息
EntryPosition position = findStartPosition(erosaConnection);

进入方法：看下里面怎么判断最后位置的 gtid判断，

protected EntryPosition findStartPosition(ErosaConnection connection) throws IOException {
    if (isGTIDMode()) {
        // GTID模式下，CanalLogPositionManager里取最后的gtid，没有则取instanc配置中的
        LogPosition logPosition = getLogPositionManager().getLatestIndexBy(destination);
        if (logPosition != null) {
            return logPosition.getPostion();
        }

        if (StringUtils.isNotEmpty(masterPosition.getGtid())) {
            return masterPosition;
        }
    }

    EntryPosition startPosition = findStartPositionInternal(connection);
    if (needTransactionPosition.get()) {
        logger.warn("prepare to find last position : {}", startPosition.toString());
        Long preTransactionStartPosition = findTransactionBeginPosition(connection, startPosition);
        if (!preTransactionStartPosition.equals(startPosition.getPosition())) {
            logger.warn("find new start Transaction Position , old : {} , new : {}",
                startPosition.getPosition(),
                preTransactionStartPosition);
            startPosition.setPosition(preTransactionStartPosition);
        }
        needTransactionPosition.compareAndSet(true, false);
    }
    return startPosition;
}

从缓存 中区 logpostion，若没有，去数据库执行，show maser status，然后是对binlog文件名字 时间戳的一些校验

protected EntryPosition findStartPositionInternal(ErosaConnection connection) {
    MysqlConnection mysqlConnection = (MysqlConnection) connection;
    LogPosition logPosition = logPositionManager.getLatestIndexBy(destination);
    if (logPosition == null) {// 找不到历史成功记录
        EntryPosition entryPosition = null;
        if (masterInfo != null && mysqlConnection.getConnector().getAddress().equals(masterInfo.getAddress())) {
            entryPosition = masterPosition;
        } else if (standbyInfo != null
                   && mysqlConnection.getConnector().getAddress().equals(standbyInfo.getAddress())) {
            entryPosition = standbyPosition;
        }

        if (entryPosition == null) {
            entryPosition = findEndPositionWithMasterIdAndTimestamp(mysqlConnection); // 默认从当前最后一个位置进行消费
        }

        // 判断一下是否需要按时间订阅
        if (StringUtils.isEmpty(entryPosition.getJournalName())) {
            // 如果没有指定binlogName，尝试按照timestamp进行查找
            if (entryPosition.getTimestamp() != null && entryPosition.getTimestamp() > 0L) {
                logger.warn("prepare to find start position {}:{}:{}",
                    new Object[] { "", "", entryPosition.getTimestamp() });
                return findByStartTimeStamp(mysqlConnection, entryPosition.getTimestamp());
            } else {
                logger.warn("prepare to find start position just show master status");
                return findEndPositionWithMasterIdAndTimestamp(mysqlConnection); // 默认从当前最后一个位置进行消费
            }
        } else {
            if (entryPosition.getPosition() != null && entryPosition.getPosition() > 0L) {
                // 如果指定binlogName + offest，直接返回
                entryPosition = findPositionWithMasterIdAndTimestamp(mysqlConnection, entryPosition);
                logger.warn("prepare to find start position {}:{}:{}",
                    new Object[] { entryPosition.getJournalName(), entryPosition.getPosition(),
                            entryPosition.getTimestamp() });
                return entryPosition;
            } else {
                EntryPosition specificLogFilePosition = null;
                if (entryPosition.getTimestamp() != null && entryPosition.getTimestamp() > 0L) {
                    // 如果指定binlogName +
                    // timestamp，但没有指定对应的offest，尝试根据时间找一下offest
                    EntryPosition endPosition = findEndPosition(mysqlConnection);
                    if (endPosition != null) {
                        logger.warn("prepare to find start position {}:{}:{}",
                            new Object[] { entryPosition.getJournalName(), "", entryPosition.getTimestamp() });
                        specificLogFilePosition = findAsPerTimestampInSpecificLogFile(mysqlConnection,
                            entryPosition.getTimestamp(),
                            endPosition,
                            entryPosition.getJournalName(),
                            true);
                    }
                }

                if (specificLogFilePosition == null) {
                    // position不存在，从文件头开始
                    entryPosition.setPosition(BINLOG_START_OFFEST);
                    return entryPosition;
                } else {
                    return specificLogFilePosition;
                }
            }
        }
    } else {
        if (logPosition.getIdentity().getSourceAddress().equals(mysqlConnection.getConnector().getAddress())) {
            if (dumpErrorCountThreshold >= 0 && dumpErrorCount > dumpErrorCountThreshold) {
                // binlog定位位点失败,可能有两个原因:
                // 1. binlog位点被删除
                // 2.vip模式的mysql,发生了主备切换,判断一下serverId是否变化,针对这种模式可以发起一次基于时间戳查找合适的binlog位点
                boolean case2 = (standbyInfo == null || standbyInfo.getAddress() == null)
                                && logPosition.getPostion().getServerId() != null
                                && !logPosition.getPostion().getServerId().equals(findServerId(mysqlConnection));
                if (case2) {
                    long timestamp = logPosition.getPostion().getTimestamp();
                    long newStartTimestamp = timestamp - fallbackIntervalInSeconds * 1000;
                    logger.warn("prepare to find start position by last position {}:{}:{}", new Object[] { "", "",
                            logPosition.getPostion().getTimestamp() });
                    EntryPosition findPosition = findByStartTimeStamp(mysqlConnection, newStartTimestamp);
                    // 重新置为一下
                    dumpErrorCount = 0;
                    return findPosition;
                }
            }
            // 其余情况
            logger.warn("prepare to find start position just last position\n {}",
                JsonUtils.marshalToString(logPosition));
            return logPosition.getPostion();
        } else {
            // 针对切换的情况，考虑回退时间
            long newStartTimestamp = logPosition.getPostion().getTimestamp() - fallbackIntervalInSeconds * 1000;
            logger.warn("prepare to find start position by switch {}:{}:{}", new Object[] { "", "",
                    logPosition.getPostion().getTimestamp() });
            return findByStartTimeStamp(mysqlConnection, newStartTimestamp);
        }
    }
}

解析以上代码：

（1） 

public LogPosition getLatestIndexBy(String destination) {
    LogPosition logPosition = primary.getLatestIndexBy(destination);
    if (logPosition != null) {
        return logPosition;
    }
    return secondary.getLatestIndexBy(destination);
}

public LogPosition getLatestIndexBy(String destination) {
    return positions.get(destination);
}

从内存positons这个map里获取destination ，若获取不到进入下面这个方法

public LogPosition getLatestIndexBy(String destination) {
    List clientIdentities = metaManager.listAllSubscribeInfo(destination);
    LogPosition result = null;
    if (!CollectionUtils.isEmpty(clientIdentities)) {
        // 尝试找到一个最小的logPosition
        for (ClientIdentity clientIdentity : clientIdentities) {
            LogPosition position = (LogPosition) metaManager.getCursor(clientIdentity);
            if (position == null) {
                continue;
            }

            if (result == null) {
                result = position;
            } else {
                result = CanalEventUtils.min(result, position);
            }
        }
    }

    return result;
}

 size = 1

判断配置文件中的主库信息是否与当前的数据库连接connection的地址一致，如果一致，如果一致，那么直接取properties文件中的master的位点信息，如果主库不一致，那么判断从库standby的connection地址，如果是从库，那么直接取从库的位点信息

若都不是：那么系统默认从当前时间开始消费。

protected EntryPosition findEndPositionWithMasterIdAndTimestamp(MysqlConnection connection) {
    MysqlConnection mysqlConnection = (MysqlConnection) connection;
    final EntryPosition endPosition = findEndPosition(mysqlConnection);
    if (tableMetaTSDB != null) {
        long startTimestamp = System.currentTimeMillis();
        return findAsPerTimestampInSpecificLogFile(mysqlConnection,
            startTimestamp,
            endPosition,
            endPosition.getJournalName(),
            true);
    } else {
        return endPosition;
    }
}

然后去数据库执行show master status

获得最新的位点和时间戳 

/**
 * 根据给定的时间戳，在指定的binlog中找到最接近于该时间戳(必须是小于时间戳)的一个事务起始位置。
 * 针对最后一个binlog会给定endPosition，避免无尽的查询
 */
private EntryPosition findAsPerTimestampInSpecificLogFile(MysqlConnection mysqlConnection,
                                                          final Long startTimestamp,
                                                          final EntryPosition endPosition,
                                                          final String searchBinlogFile,
                                                          final Boolean justForPositionTimestamp) {

    final LogPosition logPosition = new LogPosition();
    try {
        mysqlConnection.reconnect();
        // 开始遍历文件
        mysqlConnection.seek(searchBinlogFile, 4L, new SinkFunction() {

            private LogPosition lastPosition;

            public boolean sink(LogEvent event) {
                EntryPosition entryPosition = null;
                try {
                    CanalEntry.Entry entry = parseAndProfilingIfNecessary(event, true);
                    if (justForPositionTimestamp && logPosition.getPostion() == null && event.getWhen() > 0) {
                        // 初始位点
                        entryPosition = new EntryPosition(searchBinlogFile,
                            event.getLogPos(),
                            event.getWhen() * 1000,
                            event.getServerId());
                        logPosition.setPostion(entryPosition);
                    }

                    if (entry == null) {
                        return true;
                    }

                    String logfilename = entry.getHeader().getLogfileName();
                    Long logfileoffset = entry.getHeader().getLogfileOffset();
                    Long logposTimestamp = entry.getHeader().getExecuteTime();
                    Long serverId = entry.getHeader().getServerId();

                    if (CanalEntry.EntryType.TRANSACTIONBEGIN.equals(entry.getEntryType())
                        || CanalEntry.EntryType.TRANSACTIONEND.equals(entry.getEntryType())) {
                        if (logger.isDebugEnabled()) {
                            logger.debug("compare exit condition:{},{},{}, startTimestamp={}...", new Object[] {
                                    logfilename, logfileoffset, logposTimestamp, startTimestamp });
                        }
                        // 事务头和尾寻找第一条记录时间戳，如果最小的一条记录都不满足条件，可直接退出
                        if (logposTimestamp >= startTimestamp) {
                            return false;
                        }
                    }

                    if (StringUtils.equals(endPosition.getJournalName(), logfilename)
                        && endPosition.getPosition() < logfileoffset) {
                        return false;
                    }

                    // 记录一下上一个事务结束的位置，即下一个事务的position
                    // position = current +
                    // data.length，代表该事务的下一条offest，避免多余的事务重复
                    if (CanalEntry.EntryType.TRANSACTIONEND.equals(entry.getEntryType())) {
                        entryPosition = new EntryPosition(logfilename, logfileoffset, logposTimestamp, serverId);
                        if (logger.isDebugEnabled()) {
                            logger.debug("set {} to be pending start position before finding another proper one...",
                                entryPosition);
                        }
                        logPosition.setPostion(entryPosition);
                    } else if (CanalEntry.EntryType.TRANSACTIONBEGIN.equals(entry.getEntryType())) {
                        // 当前事务开始位点
                        entryPosition = new EntryPosition(logfilename, logfileoffset, logposTimestamp, serverId);
                        if (logger.isDebugEnabled()) {
                            logger.debug("set {} to be pending start position before finding another proper one...",
                                entryPosition);
                        }
                        logPosition.setPostion(entryPosition);
                    }

                    lastPosition = buildLastPosition(entry);
                } catch (Throwable e) {
                    processSinkError(e, lastPosition, searchBinlogFile, 4L);
                }

                return running;
            }
        });

    } catch (IOException e) {
        logger.error("ERROR ## findAsPerTimestampInSpecificLogFile has an error", e);
    }

    if (logPosition.getPostion() != null) {
        return logPosition.getPostion();
    } else {
        return null;
    }
}

开始解析二进制，header body，转换，怎么转换的，下一篇开始

然后最终如下日志：

1.6）启动后parse解析（暂时不会走着）

afterStartEventParser(eventParser);

protected void afterStartEventParser(CanalEventParser eventParser) {
    // 读取一下历史订阅的filter信息
    List clientIdentitys = metaManager.listAllSubscribeInfo(destination);
    for (ClientIdentity clientIdentity : clientIdentitys) {
        subscribeChange(clientIdentity);
    }
}

返回true

没有做过滤

扫描配置项：（暂时不会走着）

接口类两个实现，目前manager里面为空实现

把action和des放到map里

public void register(String destination, InstanceAction action) {
    if (action != null) {
        actions.put(destination, action);
    } else {
        actions.put(destination, defaultAction);
    }
}

全局变量，定时扫描配置的任务 5秒

if (autoScan) {
    instanceConfigMonitors.get(globalInstanceConfig.getMode()).start();
    for (InstanceConfigMonitor monitor : instanceConfigMonitors.values()) {
        if (!monitor.isStart()) {
            monitor.start();
        }
    }
}

1.7 通信方面了，数据开始流转的地方

// 启动网络接口
canalServer.start();

1.8netty 相关

public void start() {
    super.start();

    if (!embeddedServer.isStart()) {
        embeddedServer.start();
    }

    this.bootstrap = new ServerBootstrap(new NioServerSocketChannelFactory(Executors.newCachedThreadPool(),
        Executors.newCachedThreadPool()));
    /*
     * enable keep-alive mechanism, handle abnormal network connection
     * scenarios on OS level. the threshold parameters are depended on OS.
     * e.g. On Linux: net.ipv4.tcp_keepalive_time = 300
     * net.ipv4.tcp_keepalive_probes = 2 net.ipv4.tcp_keepalive_intvl = 30
     */
    bootstrap.setOption("child.keepAlive", true);
    /*
     * optional parameter.
     */
    bootstrap.setOption("child.tcpNoDelay", true);

    // 构造对应的pipeline
    bootstrap.setPipelineFactory(new ChannelPipelineFactory() {

        public ChannelPipeline getPipeline() throws Exception {
            ChannelPipeline pipelines = Channels.pipeline();
            pipelines.addLast(FixedHeaderFrameDecoder.class.getName(), new FixedHeaderFrameDecoder());
            // support to maintain child socket channel.
            pipelines.addLast(HandshakeInitializationHandler.class.getName(),
                new HandshakeInitializationHandler(childGroups));
            pipelines.addLast(ClientAuthenticationHandler.class.getName(),
                new ClientAuthenticationHandler(embeddedServer));

            SessionHandler sessionHandler = new SessionHandler(embeddedServer);
            pipelines.addLast(SessionHandler.class.getName(), sessionHandler);
            return pipelines;
        }
    });

    // 启动
    if (StringUtils.isNotEmpty(ip)) {
        this.serverChannel = bootstrap.bind(new InetSocketAddress(this.ip, this.port));
    } else {
        this.serverChannel = bootstrap.bind(new InetSocketAddress(this.port));
    }
}

这里采用的是netty 4.0 版本，