【Nacos】源码之服务端AP架构集群节点数据的同步

当Nacos集群部署时，临时实例数据在集群之间是如何进行同步的？

Nacos针对临时实例数据在集群之间的同步开发了Distro一致性协议，Distro一致性协议是弱一致性协议，用来保证Nacos注册中心的可用性，当临时实例注册到Nacos注册中心时，集群的实例数据并不是一致的，当通过Distro协议同步之后才最终达到一致性，所以Distro协议保证了Nacos注册中心的AP（可用性）。

新节点同步实例数据

如果nacos集群中有新的节点加入，新节点启动时就会从其他节点进行全量拉取数据。当DistroProtocol初始化时，调用startDistroTask方法进行全量拉取数据：

com.alibaba.nacos.core.distributed.distro.DistroProtocol#startDistroTask

private void startDistroTask() {
	if (EnvUtil.getStandaloneMode()) {
		isInitialized = true;
		return;
	}
	/**
         * 开启数据校验任务
         */
	startVerifyTask();
	/**
         * 开启加载数据任务
         */
	startLoadTask();
}

开启加载全量数据的任务，提交了一个异步任务DistroLoadDataTask。
com.alibaba.nacos.core.distributed.distro.DistroProtocol#startLoadTask

private void startLoadTask() {
	DistroCallback loadCallback = new DistroCallback() {
		@Override
		public void onSuccess() {
			isInitialized = true;
		}

		@Override
		public void onFailed(Throwable throwable) {
			isInitialized = false;
		}
	};
	// 立即执行
	GlobalExecutor.submitLoadDataTask(
		new DistroLoadDataTask(memberManager, distroComponentHolder, distroConfig, loadCallback));
}

下面来看DistroLoadDataTask的run()方法。run方法使用load方法加载从远程加载全量数据，如果检测到加载数据没有完成，则继续提交全量拉取数据的任务，否则进行任务的成功回调。如果加载数据发生了异常，则进行任务的失败回调。
com.alibaba.nacos.core.distributed.distro.task.load.DistroLoadDataTask#run

public void run() {
	try {
		// 加载数据
		load();
		if (!checkCompleted()) {
			// 加载不成功，延迟加载数据
			GlobalExecutor.submitLoadDataTask(this, distroConfig.getLoadDataRetryDelayMillis());
		} else {
			loadCallback.onSuccess();
			Loggers.DISTRO.info("[DISTRO-INIT] load snapshot data success");
		}
	} catch (Exception e) {
		loadCallback.onFailed(e);
		Loggers.DISTRO.error("[DISTRO-INIT] load snapshot data failed. ", e);
	}
}

com.alibaba.nacos.core.distributed.distro.task.load.DistroLoadDataTask#load

private void load() throws Exception {
	// 在服务启动的时候，是没有其他远程服务的地址的，如果服务地址都是空的，则进行等待，直到服务地址不为空。
	while (memberManager.allMembersWithoutSelf().isEmpty()) {
		Loggers.DISTRO.info("[DISTRO-INIT] waiting server list init...");
		TimeUnit.SECONDS.sleep(1);
	}
	// 接着判断数据存储类型是否为空，如果为空，则进行等待，直到服务地址不为空。
	while (distroComponentHolder.getDataStorageTypes().isEmpty()) {
		Loggers.DISTRO.info("[DISTRO-INIT] waiting distro data storage register...");
		TimeUnit.SECONDS.sleep(1);
	}
	// 遍历所有的数据存储类型，判断loadCompletedMap是否存在数据存储类型和该类型的数据是否已经加载完成，
	// 如果没有则调用loadAllDataSnapshotFromRemote进行全量数据的加载：
	for (String each : distroComponentHolder.getDataStorageTypes()) {
		if (!loadCompletedMap.containsKey(each) || !loadCompletedMap.get(each)) {
			// 没完成的继续加载数据
			loadCompletedMap.put(each, loadAllDataSnapshotFromRemote(each));
		}
	}
}

loadAllDataSnapshotFromRemote()负责从远程服务器拉取数据。
com.alibaba.nacos.core.distributed.distro.task.load.DistroLoadDataTask#loadAllDataSnapshotFromRemote

private boolean loadAllDataSnapshotFromRemote(String resourceType) {
	DistroTransportAgent transportAgent = distroComponentHolder.findTransportAgent(resourceType);
	DistroDataProcessor dataProcessor = distroComponentHolder.findDataProcessor(resourceType);
	if (null == transportAgent || null == dataProcessor) {
		Loggers.DISTRO.warn("[DISTRO-INIT] Can't find component for type {}, transportAgent: {}, dataProcessor: {}",
							resourceType, transportAgent, dataProcessor);
		return false;
	}
	// 遍历所有的远程服务地址，除了自己
	for (Member each : memberManager.allMembersWithoutSelf()) {
		try {
			Loggers.DISTRO.info("[DISTRO-INIT] load snapshot {} from {}", resourceType, each.getAddress());
			/**
                 * 从远程获取所有的数据
                 */
			DistroData distroData = transportAgent.getDatumSnapshot(each.getAddress());
			// 处理数据
			boolean result = dataProcessor.processSnapshot(distroData);
			Loggers.DISTRO
				.info("[DISTRO-INIT] load snapshot {} from {} result: {}", resourceType, each.getAddress(),
					  result);
			if (result) {
				return true;
			}
		} catch (Exception e) {
			Loggers.DISTRO.error("[DISTRO-INIT] load snapshot {} from {} failed.", resourceType, each.getAddress(), e);
		}
	}
	return false;
}

loadAllDataSnapshotFromRemote方法做了两件事：

1. 通过http请求拉取远程服务的所有全量数据：拉取数据的接口为：/distro/v1/ns/distro/datums
2. 处理拉取回来的全量数据

处理全量数据的方法为processData()：
com.alibaba.nacos.naming.consistency.ephemeral.distro.DistroConsistencyServiceImpl#processData(byte[])

private boolean processData(byte[] data) throws Exception {
	if (data.length > 0) {
		// 反序列化数据
		Map<String, Datum<Instances>> datumMap = serializer.deserializeMap(data, Instances.class);

		for (Map.Entry<String, Datum<Instances>> entry : datumMap.entrySet()) {
			dataStore.put(entry.getKey(), entry.getValue());

			if (!listeners.containsKey(entry.getKey())) {
				// pretty sure the service not exist:
				if (switchDomain.isDefaultInstanceEphemeral()) {
					// create empty service
					Loggers.DISTRO.info("creating service {}", entry.getKey());
					Service service = new Service();
					String serviceName = KeyBuilder.getServiceName(entry.getKey());
					String namespaceId = KeyBuilder.getNamespace(entry.getKey());
					service.setName(serviceName);
					service.setNamespaceId(namespaceId);
					service.setGroupName(Constants.DEFAULT_GROUP);
					// now validate the service. if failed, exception will be thrown
					service.setLastModifiedMillis(System.currentTimeMillis());
					service.recalculateChecksum();

					// The Listener corresponding to the key value must not be empty
					RecordListener listener = listeners.get(KeyBuilder.SERVICE_META_KEY_PREFIX).peek();
					if (Objects.isNull(listener)) {
						return false;
					}
					listener.onChange(KeyBuilder.buildServiceMetaKey(namespaceId, serviceName), service);
				}
			}
		}

		for (Map.Entry<String, Datum<Instances>> entry : datumMap.entrySet()) {

			if (!listeners.containsKey(entry.getKey())) {
				// Should not happen:
				Loggers.DISTRO.warn("listener of {} not found.", entry.getKey());
				continue;
			}

			try {
				for (RecordListener listener : listeners.get(entry.getKey())) {
					/**
                         * @see Service#onChange(java.lang.String, com.alibaba.nacos.naming.core.Instances)
                         */
					listener.onChange(entry.getKey(), entry.getValue().value);
				}
			} catch (Exception e) {
				Loggers.DISTRO.error("[NACOS-DISTRO] error while execute listener of key: {}", entry.getKey(), e);
				continue;
			}

			// Update data store if listener executed successfully:
			dataStore.put(entry.getKey(), entry.getValue());
		}
	}
	return true;
}

最后会调用到Service.onChange()方法，与实例的注册一样调用此方法更新注册表。

再来看看远程服务是如何处理全量拉取数据的请求的：
com.alibaba.nacos.naming.controllers.DistroController#getAllDatums

@GetMapping("/datums")
public ResponseEntity getAllDatums() {
	// 服务端查询所有实例数据的入口
	DistroData distroData = distroProtocol.onSnapshot(KeyBuilder.INSTANCE_LIST_KEY_PREFIX);
	return ResponseEntity.ok(distroData.getContent());
}

com.alibaba.nacos.core.distributed.distro.DistroProtocol#onSnapshot

public DistroData onSnapshot(String type) {
	DistroDataStorage distroDataStorage = distroComponentHolder.findDataStorage(type);
	if (null == distroDataStorage) {
		Loggers.DISTRO.warn("[DISTRO] Can't find data storage for received key {}", type);
		return new DistroData(new DistroKey("snapshot", type), new byte[0]);
	}
	// 查询所有的实例数据
	return distroDataStorage.getDatumSnapshot();
}

com.alibaba.nacos.naming.consistency.ephemeral.distro.component.DistroDataStorageImpl#getDatumSnapshot

public DistroData getDatumSnapshot() {
	// 从缓存中获取所有的实例数据
	Map<String, Datum> result = dataStore.getDataMap();
	byte[] dataContent = ApplicationUtils.getBean(Serializer.class).serialize(result);
	DistroKey distroKey = new DistroKey("snapshot", KeyBuilder.INSTANCE_LIST_KEY_PREFIX);
	return new DistroData(distroKey, dataContent);
}

全量数据拉取无非就是从内存dataStore中获取所有临时实例的数据，并且对数据进行序列化，然后返回给客户端。

数据校验任务

Nacos AP集群为保证数据的最终一致性会开启一个数据校验的定时任务来检查各个节点之间的数据是否一致，不一致就会进行数据的同步更新。

数据校验任务DistroVerifyTask与同步全量数据任务DistroLoadDataTask同样是在DistroProtocol实例化是创建的。

com.alibaba.nacos.core.distributed.distro.DistroProtocol#startDistroTask

private void startDistroTask() {
	if (EnvUtil.getStandaloneMode()) {
		isInitialized = true;
		return;
	}
	/**
         * 开启数据校验任务
         */
	startVerifyTask();
	/**
         * 开启加载数据任务
         */
	startLoadTask();
}

private void startVerifyTask() {
	// 5s执行一次数据校验任务
	GlobalExecutor.schedulePartitionDataTimedSync(new DistroVerifyTask(memberManager, distroComponentHolder),
												  distroConfig.getVerifyIntervalMillis());
}

数据校验任务默认5s执行一次，会将缓存中所有的key调用接口/nacos/v1/ns/distro/checksum发送给远程服务器。
com.alibaba.nacos.core.distributed.distro.task.verify.DistroVerifyTask#run

public void run() {
	try {
		// 获取集群中的其他节点
		List<Member> targetServer = serverMemberManager.allMembersWithoutSelf();
		if (Loggers.DISTRO.isDebugEnabled()) {
			Loggers.DISTRO.debug("server list is: {}", targetServer);
		}
		for (String each : distroComponentHolder.getDataStorageTypes()) {
			// 校验数据
			verifyForDataStorage(each, targetServer);
		}
	} catch (Exception e) {
		Loggers.DISTRO.error("[DISTRO-FAILED] verify task failed.", e);
	}
}

private void verifyForDataStorage(String type, List<Member> targetServer) {
	/**
         * @see com.alibaba.nacos.naming.consistency.ephemeral.distro.component.DistroDataStorageImpl#getVerifyData()
         */
	DistroData distroData = distroComponentHolder.findDataStorage(type).getVerifyData();
	if (null == distroData) {
		return;
	}
	distroData.setType(DataOperation.VERIFY);
	// 遍历集群中的其他节点
	for (Member member : targetServer) {
		try {
			// 发送数据校验的请求
			distroComponentHolder.findTransportAgent(type).syncVerifyData(distroData, member.getAddress());
		} catch (Exception e) {
			Loggers.DISTRO.error(String
								 .format("[DISTRO-FAILED] verify data for type %s to %s failed.", type, member.getAddress()), e);
		}
	}
}

接下来看远程服务器端接受到数据校验任务的请求时是怎么处理的：

com.alibaba.nacos.naming.controllers.DistroController#syncChecksum

@PutMapping("/checksum")
public ResponseEntity syncChecksum(@RequestParam String source, @RequestBody Map<String, String> dataMap) {
	// 收到校验数据的请求
	DistroHttpData distroHttpData = new DistroHttpData(createDistroKey(source), dataMap);
	// 开始校验
	distroProtocol.onVerify(distroHttpData);
	return ResponseEntity.ok("ok");
}

com.alibaba.nacos.core.distributed.distro.DistroProtocol#onVerify

public boolean onVerify(DistroData distroData) {
	String resourceType = distroData.getDistroKey().getResourceType();
	DistroDataProcessor dataProcessor = distroComponentHolder.findDataProcessor(resourceType);
	if (null == dataProcessor) {
		Loggers.DISTRO.warn("[DISTRO] Can't find verify data process for received data {}", resourceType);
		return false;
	}
	/**
         * 处理校验数据
         */
	return dataProcessor.processVerifyData(distroData);
}

com.alibaba.nacos.naming.consistency.ephemeral.distro.DistroConsistencyServiceImpl#processVerifyData

public boolean processVerifyData(DistroData distroData) {
	DistroHttpData distroHttpData = (DistroHttpData) distroData;
	String sourceServer = distroData.getDistroKey().getResourceKey();
	Map<String, String> verifyData = (Map<String, String>) distroHttpData.getDeserializedContent();
	// 校验数据
	onReceiveChecksums(verifyData, sourceServer);
	return true;
}

com.alibaba.nacos.naming.consistency.ephemeral.distro.DistroConsistencyServiceImpl#onReceiveChecksums

public void onReceiveChecksums(Map<String, String> checksumMap, String server) {

	if (syncChecksumTasks.containsKey(server)) {
		// Already in process of this server:
		Loggers.DISTRO.warn("sync checksum task already in process with {}", server);
		return;
	}

	syncChecksumTasks.put(server, "1");

	try {

		// 保存要更新的key
		List<String> toUpdateKeys = new ArrayList<>();
		// 保存要删除的key
		List<String> toRemoveKeys = new ArrayList<>();
		for (Map.Entry<String, String> entry : checksumMap.entrySet()) {
			if (distroMapper.responsible(KeyBuilder.getServiceName(entry.getKey()))) {
				// this key should not be sent from remote server:
				Loggers.DISTRO.error("receive responsible key timestamp of " + entry.getKey() + " from " + server);
				// abort the procedure:
				return;
			}

			if (!dataStore.contains(entry.getKey()) || dataStore.get(entry.getKey()).value == null || !dataStore
				.get(entry.getKey()).value.getChecksum().equals(entry.getValue())) {
				toUpdateKeys.add(entry.getKey());
			}
		}

		for (String key : dataStore.keys()) {

			if (!server.equals(distroMapper.mapSrv(KeyBuilder.getServiceName(key)))) {
				continue;
			}

			if (!checksumMap.containsKey(key)) {
				toRemoveKeys.add(key);
			}
		}

		Loggers.DISTRO
			.info("to remove keys: {}, to update keys: {}, source: {}", toRemoveKeys, toUpdateKeys, server);

		for (String key : toRemoveKeys) {
			// 删除实例
			onRemove(key);
		}

		if (toUpdateKeys.isEmpty()) {
			// 没有要更新的key就返回了
			// 说明两个服务之间的实例数据一致
			return;
		}

		try {
			DistroHttpCombinedKey distroKey = new DistroHttpCombinedKey(KeyBuilder.INSTANCE_LIST_KEY_PREFIX,
																		server);
			distroKey.getActualResourceTypes().addAll(toUpdateKeys);
			// 从其他节点获取要更新的key对应的数据
			// 调用/nacos/v1/ns/distro/datum
			DistroData remoteData = distroProtocol.queryFromRemote(distroKey);
			if (null != remoteData) {
				// 将数据放入缓存
				processData(remoteData.getContent());
			}
		} catch (Exception e) {
			Loggers.DISTRO.error("get data from " + server + " failed!", e);
		}
	} finally {
		// Remove this 'in process' flag:
		syncChecksumTasks.remove(server);
	}
}

处理逻辑主要是拿到请求中的所有key与本地缓存中的key进行对比，如果有不相同的key就删除，如果有新增或要更新的key就根据key去发送请求的服务器端查询然后更新本地缓存和注册表。

同步实例数据给集群其他节点

当有新的客户端注册到Nacos集群中的一个节点时，这个节点就需要将新的实例数据同步给其他节点。

遍历所有的远程服务添加DistroDelayTask任务。
com.alibaba.nacos.core.distributed.distro.DistroProtocol#sync(com.alibaba.nacos.core.distributed.distro.entity.DistroKey, com.alibaba.nacos.consistency.DataOperation, long)

public void sync(DistroKey distroKey, DataOperation action, long delay) {
	// 遍历其他节点，不包括自己
	for (Member each : memberManager.allMembersWithoutSelf()) {
		DistroKey distroKeyWithTarget = new DistroKey(distroKey.getResourceKey(), distroKey.getResourceType(),
													  each.getAddress());
		DistroDelayTask distroDelayTask = new DistroDelayTask(distroKeyWithTarget, action, delay);
		/**
             * @see NacosDelayTaskExecuteEngine#addTask(java.lang.Object, com.alibaba.nacos.common.task.AbstractDelayTask)
             */
		// 添加任务
		distroTaskEngineHolder.getDelayTaskExecuteEngine().addTask(distroKeyWithTarget, distroDelayTask);
		if (Loggers.DISTRO.isDebugEnabled()) {
			Loggers.DISTRO.debug("[DISTRO-SCHEDULE] {} to {}", distroKey, each.getAddress());
		}
	}
}

DistroDelayTask任务的执行过程中又会添加一个DistroSyncChangeTask任务。
com.alibaba.nacos.core.distributed.distro.task.delay.DistroDelayTaskProcessor#process

public boolean process(NacosTask task) {
	if (!(task instanceof DistroDelayTask)) {
		return true;
	}
	DistroDelayTask distroDelayTask = (DistroDelayTask) task;
	DistroKey distroKey = distroDelayTask.getDistroKey();
	if (DataOperation.CHANGE.equals(distroDelayTask.getAction())) {
		// 又来一个任务，又异步
		DistroSyncChangeTask syncChangeTask = new DistroSyncChangeTask(distroKey, distroComponentHolder);
		/**
             * @see NacosExecuteTaskExecuteEngine#addTask(java.lang.Object, com.alibaba.nacos.common.task.AbstractExecuteTask)
             * @see DistroSyncChangeTask#run()
             */
		distroTaskEngineHolder.getExecuteWorkersManager().addTask(distroKey, syncChangeTask);
		return true;
	}
	return false;
}

DistroSyncChangeTask任务的执行过程中会调用远程服务器接口/nacos/v1/ns/distro/datum进行数据的同步。
com.alibaba.nacos.core.distributed.distro.task.execute.DistroSyncChangeTask#run

public void run() {
	Loggers.DISTRO.info("[DISTRO-START] {}", toString());
	try {
		String type = getDistroKey().getResourceType();
		DistroData distroData = distroComponentHolder.findDataStorage(type).getDistroData(getDistroKey());
		distroData.setType(DataOperation.CHANGE);
		/**
             * @see com.alibaba.nacos.naming.consistency.ephemeral.distro.component.DistroHttpAgent#syncData(DistroData, String)
             */
		// 调用http接口 put /nacos/v1/ns/distro/datum
		boolean result = distroComponentHolder.findTransportAgent(type).syncData(distroData, getDistroKey().getTargetServer());
		if (!result) {
			handleFailedTask();
		}
		Loggers.DISTRO.info("[DISTRO-END] {} result: {}", toString(), result);
	} catch (Exception e) {
		Loggers.DISTRO.warn("[DISTRO] Sync data change failed.", e);
		handleFailedTask();
	}
}

下面看看远程服务器的接口/nacos/v1/ns/distro/datum收到请求时怎么处理：

com.alibaba.nacos.naming.controllers.DistroController#onSyncDatum

@PutMapping("/datum")
public ResponseEntity onSyncDatum(@RequestBody Map<String, Datum<Instances>> dataMap) throws Exception {

	// 同步数据的入口
	if (dataMap.isEmpty()) {
		Loggers.DISTRO.error("[onSync] receive empty entity!");
		throw new NacosException(NacosException.INVALID_PARAM, "receive empty entity!");
	}

	for (Map.Entry<String, Datum<Instances>> entry : dataMap.entrySet()) {
		if (KeyBuilder.matchEphemeralInstanceListKey(entry.getKey())) {
			String namespaceId = KeyBuilder.getNamespace(entry.getKey());
			String serviceName = KeyBuilder.getServiceName(entry.getKey());
			if (!serviceManager.containService(namespaceId, serviceName) && switchDomain
				.isDefaultInstanceEphemeral()) {
				serviceManager.createEmptyService(namespaceId, serviceName, true);
			}
			DistroHttpData distroHttpData = new DistroHttpData(createDistroKey(entry.getKey()), entry.getValue());
			// 收到数据进行处理
			distroProtocol.onReceive(distroHttpData);
		}
	}
	return ResponseEntity.ok("ok");
}

com.alibaba.nacos.core.distributed.distro.DistroProtocol#onReceive

public boolean onReceive(DistroData distroData) {
	String resourceType = distroData.getDistroKey().getResourceType();
	DistroDataProcessor dataProcessor = distroComponentHolder.findDataProcessor(resourceType);
	if (null == dataProcessor) {
		Loggers.DISTRO.warn("[DISTRO] Can't find data process for received data {}", resourceType);
		return false;
	}
	return dataProcessor.processData(distroData);
}

com.alibaba.nacos.naming.consistency.ephemeral.distro.DistroConsistencyServiceImpl#processData(com.alibaba.nacos.core.distributed.distro.entity.DistroData)

public boolean processData(DistroData distroData) {
	DistroHttpData distroHttpData = (DistroHttpData) distroData;
	Datum<Instances> datum = (Datum<Instances>) distroHttpData.getDeserializedContent();
	// 异步处理数据
	onPut(datum.key, datum.value);
	return true;
}

processData()方法将同步过来的数据进行反序列化，然后调用onPut()方法进行临时数据缓存并添加实例变更的任务，后续逻辑与实例注册后的处理逻辑一致。