文前说明
作为码农中的一员,需要不断的学习,我工作之余将一些分析总结和学习笔记写成博客与大家一起交流,也希望采用这种方式记录自己的学习之旅。
本文仅供学习交流使用,侵权必删。
不用于商业目的,转载请注明出处。
分析整理的版本为 Ovirt 3.4.5 版本。
主机
ResourceManager
- engine 将 InitBackendServicesOnStartupBean 交由 EJB 容器进行管理,注解为 @Singleton 和 @Startup,在应用启动前,对 InitBackendServicesOnStartupBean 进行初始化,并且实例化 ResourceManager。
@Singleton
@Startup
@DependsOn({ "Backend"})
public class InitBackendServicesOnStartupBean implements InitBackendServicesOnStartup{
......
@Override
@PostConstruct
public void create() {
......
ResourceManager.getInstance().init();
......
- ResourceManager 实例化,其中最重要的操作如下:
for (VDS curVds : allVdsList) {
AddVds(curVds, true);
}
IrsBrokerCommand.init();
AddVds 根据数据库中的主机,初始化一个 vdsManager,而这个 vdsManager 用于管理所有的主机。vdsManager 主要建立了与 vdsm 的 xml-rpc 连接。
VdsManager vdsManager = VdsManager.buildVdsManager(vds);
private void InitVdsBroker() {
log.infoFormat("Initialize vdsBroker ({0},{1})", _vds.getHostName(), _vds.getPort());
// Get the values of the timeouts:
int clientTimeOut = Config. getValue(ConfigValues.vdsTimeout) * 1000;
int connectionTimeOut = Config.getValue(ConfigValues.vdsConnectionTimeout) * 1000;
int clientRetries = Config.getValue(ConfigValues.vdsRetries);
Pair returnValue =
XmlRpcUtils.getConnection(_vds.getHostName(),
_vds.getPort(),
clientTimeOut,
connectionTimeOut,
clientRetries,
VdsServerConnector.class,
Config. getValue(ConfigValues.EncryptHostCommunication));
_vdsProxy = new VdsServerWrapper(returnValue.getFirst(), returnValue.getSecond());
}
- IrsBrokerCommand.init() 初始化所有数据中心的 IrsProxyData,也就是 SPM。管理每个数据中心的资源,如存储域,加入到 _irsProxyData 的缓存中,IrsProxyData 的构造方法中启动了定时任务 _updatingTimer_Elapsed 用来管理对应数据中心的主存储域以及其它的存储域。
public static void init() {
for (StoragePool sp : DbFacade.getInstance().getStoragePoolDao().getAll()) {
if (!_irsProxyData.containsKey(sp.getId())) {
_irsProxyData.put(sp.getId(), new IrsProxyData(sp.getId()));
}
}
}
public IrsProxyData(Guid storagePoolId) {
_storagePoolId = storagePoolId;
int storagePoolRefreshTime = Config. getValue(ConfigValues.StoragePoolRefreshTimeInSeconds);
storagePoolRefreshJobId = SchedulerUtilQuartzImpl.getInstance().scheduleAFixedDelayJob(this,
"_updatingTimer_Elapsed", new Class[0], new Object[0], storagePoolRefreshTime,
storagePoolRefreshTime, TimeUnit.SECONDS);
}
@OnTimerMethodAnnotation("_updatingTimer_Elapsed")
public void _updatingTimer_Elapsed() {
try {
synchronized (syncObj) {
if (!_disposed) {
StoragePool storagePool = DbFacade.getInstance().getStoragePoolDao()
.get(_storagePoolId);
if (storagePool != null) {
// when there are no hosts in status up, it means that there shouldn't be domain monitoring
// so all the domains need to move to "unknown" status as otherwise their status won't change.
if (DbFacade.getInstance()
.getVdsDao()
.getAllForStoragePoolAndStatus(_storagePoolId, reportingVdsStatus)
.isEmpty()) {
StoragePoolDomainHelper.updateApplicablePoolDomainsStatuses(_storagePoolId,
StoragePoolDomainHelper.storageDomainMonitoredStatus,
StorageDomainStatus.Unknown, "no reporting hosts");
}
if (storagePool.getStatus() == StoragePoolStatus.Up
|| storagePool.getStatus() == StoragePoolStatus.NonResponsive || storagePool
.getStatus() == StoragePoolStatus.Contend) {
proceedStoragePoolStats(storagePool);
}
}
}
}
} catch (Exception ex) {
}
}
- 负责了 _asyncRunningVms 缓存的管理。该缓存为实时状态刷新提供了数据源。
public boolean AddAsyncRunningVm(Guid vmId) {
boolean returnValue = false;
if (_asyncRunningVms.putIfAbsent(vmId, Boolean.TRUE) == null) {
returnValue = true;
}
return returnValue;
}
public void RemoveAsyncRunningVm(Guid vmId) {
_asyncRunningVms.remove(vmId);
getEventListener().removeAsyncRunningCommand(vmId);
}
public void succededToRunVm(Guid vmId, Guid vdsId) {
if (_asyncRunningVms.containsKey(vmId)) {
getEventListener().runningSucceded(vmId);
}
RemoveAsyncRunningVm(vmId);
}
......
- 根据 gwt-rpc 传递的 command 简称,通过 Class.fromName(commmandName) 创建对应的 Command 的实例,执行对应的业务逻辑。
private static final String VDSCommandPrefix = "VDSCommand";
......
private static String GetCommandTypeName(VDSCommandType command) {
String packageName = command.getPackageName();
String commandName = String.format("%s.%s%s", packageName, command, VDSCommandPrefix);
return commandName;
}
......
private VDSCommandBase
CreateCommand(
VDSCommandType commandType, P parameters) {
try {
@SuppressWarnings("unchecked")
Class> type =
(Class>) Class.forName(GetCommandTypeName(commandType));
Constructor> constructor =
ReflectionUtils.findConstructor(type, parameters.getClass());
if (constructor != null) {
return constructor.newInstance(new Object[] { parameters });
}
} catch (Exception e) {
if (e.getCause() != null) {
log.error("CreateCommand failed", e.getCause());
throw new RuntimeException(e.getCause().getMessage(), e.getCause());
}
log.error("CreateCommand failed", e);
}
return null;
}
......
VDSBrokerFrontendImpl
- Backend 中进行实例化。主要负责了 VDSCommad 的执行和异步 Command 的缓存。
_resourceManger = new VDSBrokerFrontendImpl();
public VDSReturnValue RunVdsCommand(VDSCommandType commandType, VDSParametersBase parameters) {
return VdsHandler.handleVdsResult(getResourceManager().runVdsCommand(commandType, parameters));
}
public VDSReturnValue RunAsyncVdsCommand(VDSCommandType commandType, VdsAndVmIDVDSParametersBase parameters,
IVdsAsyncCommand command) {
VDSReturnValue result = RunVdsCommand(commandType, parameters);
if (result.getSucceeded()) {
// Add async command to cached commands
IVdsAsyncCommand prevCommand = _asyncRunningCommands.put(parameters.getVmId(), command);
if (prevCommand != null && !prevCommand.equals(command)) {
prevCommand.reportCompleted();
}
} else {
throw new VdcBLLException(result.getVdsError().getCode(), result.getExceptionString());
}
return result;
}
虚拟机
平台中,虚拟机的状态共 18 种,对应着虚拟机当前的状态,而状态是根据用户的操作以及任务的执行以及 onTimer 中定时任刷新底层 vdsm 的虚拟机状态而后做一些判断共同决定的。
虚拟机对应的操作为开机,关机,断电,休眠,迁移,重启,快照,还有一些涉及磁盘的操作,对应的业务有导入,导出以及创建磁盘等。
平台可以通过界面用户触发或者 API 调用的方式进行虚拟机操作。最终都是通过相应的 command 实现业务逻辑。
业务逻辑中:
- 进行参数、权限、资源的检测与组装。
- 通过 xml-rpc 调用 vdsm 接口,由其完成与 libvirt 的交互达到管理 qemu 进程,最终操作 KVM 完成虚拟机操作。
运行虚拟机
运行虚拟机,可以分为多种情况,一般是普通运行,还有用于安装系统的只运行一次,运行休眠的虚拟机,以及无状态模式运行也就是运行池中的虚拟机。
普通运行,通过 RunVmCommand 实现。
只运行一次,通过 RunVmOnceCommand 实现,该模式主要用于安装系统时使用,配置有启动项,cd、floppy 等,开机后配置不保存。
无状态模式启动,同样使用了 RunVmCommand 实现,通过传递参数决定是否以无状态模式启动。无状态模式启动,在开机前会创建快照,关机后会恢复该快照(或者下次开机前恢复),保持开机前的状态。
虚拟机的运行,会根据集群策略,自动选择一台合适的主机运行。
- 虽然是不同的启动开机方式,但是都是调用 CreateVmVDSCommand。
VMStatus vmStatus = (VMStatus) getBackend()
.getResourceManager()
.RunAsyncVdsCommand(VDSCommandType.CreateVm, initCreateVmParams(), this).getReturnValue();
- 普通运行、只运行一次和无状态运行,最终都调用了 CreateVDSCommand。
return new CreateVDSCommand(getParameters());
CreateVDSCommand 根据不同的参数,组装 xml,通过 xml-rpc 执行 vdsm 的 create 方法。
mVmReturn = getBroker().create(createInfo);
public OneVmReturnForXmlRpc create(Map createInfo) {
try {
Map xmlRpcReturnValue = vdsServer.create(createInfo);
OneVmReturnForXmlRpc wrapper = new OneVmReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
} catch (UndeclaredThrowableException ute) {
throw new XmlRpcRunTimeException(ute);
}
}
- 主机 vdsm 执行 create 方法。/usr/share/vdsm/API.py
def create(self, vmParams):
"""
Start up a virtual machine.
:param vmParams: required and optional VM parameters.
:type vmParams: dict
"""
vmParams['vmId'] = self._UUID
try:
if vmParams.get('vmId') in self._cif.vmContainer:
self.log.warning('vm %s already exists' % vmParams['vmId'])
return errCode['exist']
if 'hiberVolHandle' in vmParams:
vmParams['restoreState'], paramFilespec = \
self._getHibernationPaths(vmParams.pop('hiberVolHandle'))
try: # restore saved vm parameters
# NOTE: pickled params override command-line params. this
# might cause problems if an upgrade took place since the
# parmas were stored.
fname = self._cif.prepareVolumePath(paramFilespec)
......
- 运行休眠的虚拟机,需要恢复内存,执行的是 ResumeVDSCommand,调用 ResumeBrokerVDSCommand ,根据实时的状态进行回调。
ResumeBrokerVDSCommand command =
new ResumeBrokerVDSCommand(parameters);
command.execute();
mVmReturn = getBroker().resume(mVmId.toString());
proceedProxyReturnValue();
public OneVmReturnForXmlRpc pause(String vmId) {
try {
Map xmlRpcReturnValue = vdsServer.pause(vmId);
OneVmReturnForXmlRpc wrapper = new OneVmReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
} catch (UndeclaredThrowableException ute) {
throw new XmlRpcRunTimeException(ute);
}
}
- 主机 vdsm 执行 cont 方法。/usr/share/vdsm/API.py
def cont(self):
v = self._cif.vmContainer.get(self._UUID)
if not v:
return errCode['noVM']
return v.cont()
关机/断电虚拟机
- 关机执行 ShutdownVmCommand,如果不能关闭,再执行断电 StopVmCommand。
if (canShutdownVm()) {
// shutting down desktop and waiting for it in a separate thread to
// become 'down':
log.infoFormat("Sending shutdown command for VM {0}.", getVmName());
int secondsToWait = getParameters().getWaitBeforeShutdown() ? Config
. getValue(ConfigValues.VmGracefulShutdownTimeout) : 0;
// sending a shutdown command to the VM:
setActionReturnValue(Backend
.getInstance()
.getResourceManager()
.RunVdsCommand(VDSCommandType.DestroyVm,
new DestroyVmVDSCommandParameters(getVdsId(), getVmId(), false, true, secondsToWait))
.getReturnValue());
}
else {
// cannot shutdown -> send a StopVm command instead ('destroy'):
// don't log -> log will appear for the StopVmCommand we are about to run:
setCommandShouldBeLogged(false);
log.infoFormat("Cannot shutdown VM {0}, status is not up. Stopping instead.", getVmName());
StopVmParameters stopVmParams = new StopVmParameters(getVmId(), StopVmTypeEnum.CANNOT_SHUTDOWN);
// stopVmParams.ParametersCurrentUser = CurrentUser;
stopVmParams.setSessionId(getParameters().getSessionId());
Backend.getInstance().runInternalAction(VdcActionType.StopVm, stopVmParams);
}
- 关机和断电最终都调用了 DestroyVDSCommand,只是传递的参数不同,最终执行 vdsm 不同的方法。关机 shutdown,断电 destroy。
if (getParameters().getGracefully()) {
status = getBroker().shutdown(getParameters().getVmId().toString(),
String.valueOf(getParameters().getSecondsToWait()),
Config. getValue(ConfigValues.VmGracefulShutdownMessage));
} else {
status = getBroker().destroy(getParameters().getVmId().toString(), getParameters().getDestroyInfo());
}
public StatusOnlyReturnForXmlRpc shutdown(String vmId, String timeout, String message) {
try {
Map xmlRpcReturnValue = vdsServer.shutdown(vmId, timeout, message);
StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
} catch (UndeclaredThrowableException ute) {
throw new XmlRpcRunTimeException(ute);
}
}
public StatusOnlyReturnForXmlRpc destroy(String vmId, Map destroyInfo) {
try {
Map xmlRpcReturnValue = vdsServer.destroy(vmId, destroyInfo);
StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
} catch (UndeclaredThrowableException ute) {
throw new XmlRpcRunTimeException(ute);
}
}
- 主机 vdsm 关闭执行 shutdown 断电执行 destroy 方法。/usr/share/vdsm/API.py
def shutdown(self, delay=None, message=None, reboot=False, timeout=None,
force=False):
"""
Shut a VM down politely.
:param message: message to be shown to guest user before shutting down
his machine.
:param delay: grace period (seconds) to let guest user close his
applications.
:param reboot: True if reboot is desired, False for shutdown
:param timeout: number of seconds to wait before trying next
shutdown/reboot method
:param force: True if shutdown/reboot desired by any means necessary
(forceful reboot/shutdown if all graceful methods fail)
"""
try:
v = self._cif.vmContainer[self._UUID]
except KeyError:
return errCode['noVM']
if not delay:
delay = config.get('vars', 'user_shutdown_timeout')
if not message:
message = USER_SHUTDOWN_MESSAGE
if not timeout:
timeout = config.getint('vars', 'sys_shutdown_timeout')
return v.shutdown(delay, message, reboot, timeout, force)
def destroy(self, hostID, deprecatedSCSIKey):
return self._irs.destroyStoragePool(self._UUID, hostID)
休眠虚拟机
- 休眠执行 HibernateVmCommand,会创建磁盘,并且保存内存数据 CreateImageVDSCommand。再调用 HibernateVDSCommand。保存虚拟机状态 SavingState 更新数据库。
Backend
.getInstance()
.getResourceManager()
.RunVdsCommand(
VDSCommandType.CreateImage,
new CreateImageVDSCommandParameters(
getVm().getStoragePoolId(),
getStorageDomainId(),
image1GroupId,
getVm().getTotalMemorySizeInBytes(),
getMemoryVolumeType(),
VolumeFormat.RAW,
hiberVol1,
""));
Backend.getInstance().getResourceManager().RunVdsCommand(VDSCommandType.Hibernate, para);
private VDSReturnValue runHibernateBrokerVDSCommand() {
HibernateBrokerVDSCommand command =
new HibernateBrokerVDSCommand(getParameters());
command.execute();
return command.getVDSReturnValue();
}
private void changeVmStatusToSavingState() {
TransactionSupport.executeInNewTransaction(
new TransactionMethod
status = getBroker().hibernate(getParameters().getVmId().toString(),
getParameters().getHibernationVolHandle(),
getParameters().getEncryptionInfo());
public StatusOnlyReturnForXmlRpc hibernate(String vmId, String hiberVolHandle, Map encryptionInfo) {
try {
Map xmlRpcReturnValue = vdsServer.hibernate(vmId, hiberVolHandle, encryptionInfo);
StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
} catch (UndeclaredThrowableException ute) {
throw new XmlRpcRunTimeException(ute);
}
}
- 主机 vdsm 休眠执行 hibernate 方法。/usr/share/vdsm/API.py
def hibernate(self, hibernationVolHandle, encryptionInfo):
"""
Hibernate a VM.
:param hiberVolHandle: opaque string, indicating the location of
hibernation images.
:param encryptionInfo: a map value indicates the vm and template encyption
password
"""
params = {'vmId': self._UUID, 'mode': 'file',
'hiberVolHandle': hibernationVolHandle,
'vmMigrateDestPw': encryptionInfo['vmSuspendPw'],
'tmMigrateDestPw': encryptionInfo['tmSuspendPw'],
'secretUUID': encryptionInfo['secretUUID']}
response = self.migrate(params)
if not response['status']['code']:
response['status']['message'] = 'Hibernation process starting'
return response
创建快照(保存内存)
- 快照的创建实际对应于底层是创建一个 volume ,执行 CreateSnapshotCommand,调用 CreateSnapshotVDSCommand。
vdsReturnValue =
Backend
.getInstance()
.getResourceManager()
.RunVdsCommand(
VDSCommandType.CreateSnapshot,
new CreateSnapshotVDSCommandParameters(getStoragePoolId(),
getDestinationStorageDomainId(),
getImageGroupId(),
getImage().getImageId(),
getDiskImage().getSize(),
mNewCreatedDiskImage.getVolumeType(),
mNewCreatedDiskImage.getVolumeFormat(),
getDiskImage().getId(),
getDestinationImageId(),
""));
uuidReturn = getIrsProxy().createVolume(getParameters().getStorageDomainId().toString(),
getParameters().getStoragePoolId().toString(),
getParameters().getImageGroupId().toString(),
(Long.valueOf(getParameters().getImageSizeInBytes())).toString(),
getParameters().getVolumeFormat().getValue(),
getParameters().getImageType().getValue(),
2,
getParameters().getNewImageID().toString(),
getParameters().getNewImageDescription(),
getParameters().getSourceImageGroupId().toString(),
getParameters().getImageId().toString());
public OneUuidReturnForXmlRpc createVolume(String sdUUID, String spUUID, String imgGUID, String size,
int volFormat, int volType, int diskType, String volUUID, String descr, String srcImgGUID, String srcVolUUID) {
Map xmlRpcReturnValue = irsServer.createVolume(sdUUID, spUUID, imgGUID, size, volFormat,
volType, diskType, volUUID, descr, srcImgGUID, srcVolUUID);
OneUuidReturnForXmlRpc wrapper = new OneUuidReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
}
- 主机 vdsm 创建卷,执行 Volume 类的 create 方法。/usr/share/vdsm/API.py
class Volume(APIBase):
ctorArgs = ['volumeID', 'storagepoolID', 'storagedomainID', 'imageID']
class Types:
UNKNOWN = storage.volume.UNKNOWN_VOL
PREALLOCATED = storage.volume.PREALLOCATED_VOL
SPARSE = storage.volume.SPARSE_VOL
class Formats:
UNKNOWN = storage.volume.UNKNOWN_FORMAT
COW = storage.volume.COW_FORMAT
RAW = storage.volume.RAW_FORMAT
class Roles:
SHARED = storage.volume.SHARED_VOL
LEAF = storage.volume.LEAF_VOL
BLANK_UUID = storage.volume.BLANK_UUID
......
def create(self, size, volFormat, preallocate, diskType, desc,
srcImgUUID, srcVolUUID):
return self._irs.createVolume(self._sdUUID, self._spUUID,
self._imgUUID, size, volFormat,
preallocate, diskType, self._UUID, desc,
srcImgUUID, srcVolUUID)
- 如果保存内存,会执行异步任务 SnapshotVDSCommand。
protected void endVmCommand() {
......
liveSnapshotSucceeded = performLiveSnapshot(createdSnapshot);
......
protected boolean performLiveSnapshot(final Snapshot snapshot) {
try {
TransactionSupport.executeInScope(TransactionScopeOption.Suppress, new TransactionMethod() {
@Override
public Void runInTransaction() {
runVdsCommand(VDSCommandType.Snapshot, buildLiveSnapshotParameters(snapshot));
return null;
}
});
} catch (VdcBLLException e) {
handleVdsLiveSnapshotFailure(e);
return false;
}
return true;
}
private StatusOnlyReturnForXmlRpc executeSnapshotVerb() {
String vmId = getParameters().getVmId().toString();
return getParameters().isMemoryVolumeExists()
? getBroker().snapshot(vmId,
createDisksMap(),
getParameters().getMemoryVolume(),
getParameters().getSnapshotInfo())
: getBroker().snapshot(vmId, createDisksMap(), getParameters().getSnapshotInfo());
}
public StatusOnlyReturnForXmlRpc snapshot(String vmId,
Map[] disks,
String memory,
Map vmEncrypParams) {
try {
Map xmlRpcReturnValue = vdsServer.snapshot(vmId, disks, memory, vmEncrypParams);
StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
} catch (UndeclaredThrowableException ute) {
throw new XmlRpcRunTimeException(ute);
}
}
- 主机 vdsm 执行 snapshot 方法。/usr/share/vdsm/API.py
def snapshot(self, snapDrives, snapMemVolHandle=None, vmEncrypParams={}):
v = self._cif.vmContainer.get(self._UUID)
if not v:
return errCode['noVM']
memoryParams = {}
if snapMemVolHandle:
memoryParams['dst'], memoryParams['dstparams'] = \
self._getHibernationPaths(snapMemVolHandle)
return v.snapshot(snapDrives, memoryParams, vmEncrypParams)
迁移虚拟机
- 迁移在虚拟机开机的情况下,从一台主机,迁移到一台新的主机。MigrateVmCommand 选择迁往的目的主机(根据集群策略)。调用 MigrateVDSCommand。改变虚拟机的状态为 MigratingFrom,更新数据库。
setActionReturnValue(Backend
.getInstance()
.getResourceManager()
.RunAsyncVdsCommand(
VDSCommandType.Migrate,
createMigrateVDSCommandParameters(),
this)
.getReturnValue());
MigrateBrokerVDSCommand> command = new MigrateBrokerVDSCommand<>(getParameters());
command.execute();
protected void executeVdsBrokerCommand() {
status = getBroker().migrate(migrationInfo);
proceedProxyReturnValue();
}
public StatusOnlyReturnForXmlRpc migrate(Map migrationInfo) {
try {
Map xmlRpcReturnValue = vdsServer.migrate(migrationInfo);
StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
} catch (UndeclaredThrowableException ute) {
throw new XmlRpcRunTimeException(ute);
}
}
- 主机 vdsm 执行 migrate 方法。/usr/share/vdsm/API.py
def migrate(self, params):
"""
Migrate a VM to a remote host.
:param params: a dictionary containing:
*dst* - remote host or hibernation image filename
*dstparams* - hibernation image filename for vdsm parameters
*mode* - ``remote``/``file``
*method* - ``online``
*downtime* - allowed down time during online migration
*dstqemu* - remote host address dedicated for migration
"""
params['vmId'] = self._UUID
self.log.debug(params)
try:
v = self._cif.vmContainer[self._UUID]
except KeyError:
return errCode['noVM']
vmParams = v.status()
if vmParams['status'] in (vmstatus.WAIT_FOR_LAUNCH, vmstatus.DOWN):
return errCode['noVM']
if params.get('mode') == 'file':
if 'dst' not in params:
params['dst'], params['dstparams'] = \
self._getHibernationPaths(params['hiberVolHandle'])
else:
params['mode'] = 'remote'
return v.migrate(params)
- 迁移会消耗一定的时间,在这个过程中,engine 会定时查询 vdsm 的虚拟机状态,从而更新 虚拟机的实时状态并进行相应的处理数据的更新。VdsManager 的 onTimer。
public void onTimer() {
......
_vdsUpdater = new VdsUpdateRunTimeInfo(VdsManager.this, _vds, monitoringStrategy) _vdsUpdater.refresh();
磁盘资源
场景 | VDS 命令 | vdsm 接口 | VolumeType | 存储域 | VolumeFormat |
---|---|---|---|---|---|
创建模板 | CopyImageVDSCommand | CopyImage | Sparse | 与虚拟机一致 | RAW→COW |
/ | / | / | 与虚拟机一致 | 与虚拟机一致 | COW |
/ | / | / | Preallocated | 与虚拟机一致 | RAW |
模版 clone 创建 vm | CopyImageVDSCommand | CopyImage | Preallocated | 与模板一致 | RAW |
模版 thin 创建 vm | CreateSnapshotVDSCommand | createVolume | Sparse | 与模板一致 | COW |
创建 vm/增加磁盘 | CreateImageVDSCommand | createVolume | Preallocated/Sparse | File | RAW |
/ | / | / | Sparse | Block | COW |
/ | / | / | Preallocated | Block | RAW |
快照创建 vm | CopyImageVDSCommand | CopyImage | 与快照一致 | 与快照一致 | COW |
/ | / | / | Sparse | Block | RAW→COW |
/ | / | / | Preallocated | 与快照一致 | RAW |
创建快照 | CreateSnapshotVDSCommand | createVolume | Sparse | 与虚拟机一致 | COW |
导入导出域中虚拟机 | CopyImageVDSCommand | copyImage | 与导出域一致 | 与导出域一致 | 与导出域一致 |
导入外部供应商 image | CopyImageVDSCommand | copyImage | Preallocated | Block | RAW |
/ | / | / | Sparse→Preallocated | Block | RAW |
/ | / | / | Sparse | File | RAW |
/ | / | / | Sparse | 与外部一致 | COW |
/ | / | / | Preallocated→Sparse | 与外部一致 | COW |
导入模版 | CopyImageVDSCommand | copyImage | Sparse | Block | RAW→COW |
/ | / | / | Sparse | File | RAW |
/ | / | / | Preallocated | 与模板一致 | 与模板一致 |
导出虚拟机 | CopyImageVDSCommand | copyImage | 与虚拟机一致 | 与虚拟机一致 | 与虚拟机一致 |
导出模版 | CopyImageVDSCommand | copyImage | 与模板一致 | 与模板一致 | 与模板一致 |
创建模板
为了适应软加密,将 VolumeType.Sparse、VolumeFormat.RAW 并且无论存储域格式,最终将 RAW 转为 COW 格式。
- 使用了 AddVmTemplateCommand→CopyImageVDSCommand,最终执行 CopyImage。
VDSReturnValue vdsReturnValue = Backend
.getInstance()
.getResourceManager()
.RunVdsCommand(
VDSCommandType.CopyImage,
new CopyImageVDSCommandParameters(storagePoolId, getParameters().getStorageDomainId(),
getParameters().getVmId(), imageGroupId, snapshotId, destinationImageGroupID,
getDestinationImageId(), StringUtils.defaultString(newImage.getDescription()), getParameters()
.getDestinationStorageDomainId(), CopyVolumeType.SharedVol, targetFormat,
newImage.getVolumeType(), getDiskImage().isWipeAfterDelete(), false));
模版 clone 创建 vm
- 使用了 CreateCloneOfTemplateCommand→CopyImageVDSCommand,最终执行 CopyImage。
vdsReturnValue = runVdsCommand(VDSCommandType.CopyImage,
copyImageParas);
模版 thin 创建 vm
- 使用了 CreateSnapshotCommand→CreateSnapshotVDSCommand,最终执行 createVolume。
vdsReturnValue =
Backend
.getInstance()
.getResourceManager()
.RunVdsCommand(
VDSCommandType.CreateSnapshot,
new CreateSnapshotVDSCommandParameters(getStoragePoolId(),
getDestinationStorageDomainId(),
getImageGroupId(),
getImage().getImageId(),
getDiskImage().getSize(),
mNewCreatedDiskImage.getVolumeType(),
mNewCreatedDiskImage.getVolumeFormat(),
getDiskImage().getId(),
getDestinationImageId(),
""));
创建 vm/增加磁盘
- 使用了 AddVmFromScratchCommand→AddImageFromScratch→CreateImageVDSCommand,最终执行 createVolume。
tmpRetValue = Backend.getInstance().runInternalAction(
VdcActionType.AddImageFromScratch,
tempVar,
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));
VDSReturnValue vdsReturnValue = runVdsCommand(
VDSCommandType.CreateImage,
new CreateImageVDSCommandParameters(getParameters().getStoragePoolId(), getParameters()
.getStorageDomainId(), getImageGroupId(), getParameters().getDiskInfo().getSize(),
getParameters().getDiskInfo().getVolumeType(), getParameters().getDiskInfo()
.getVolumeFormat(), getDestinationImageId(), ""));
快照创建 vm
- 使用了 AddVmFromSnapshotCommand→CopyImageGroupCommand→CopyImageVDSCommand,最终执行 CopyImage。
protected VdcActionType getChildActionType() {
return VdcActionType.CopyImageGroup;
}
protected VdcReturnValueBase executeChildCopyingCommand(VdcActionParametersBase parameters) {
VdcReturnValueBase result = Backend.getInstance().runInternalAction(
getChildActionType(),
parameters,
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));
return result;
}
vdsReturnValue = runVdsCommand(
VDSCommandType.CopyImage,
copyImageParas);
创建快照
- 使用了 CreateSnapshotCommand→CreateSnapshotVDSCommand,最终执行 createVolume。
vdsReturnValue =
Backend
.getInstance()
.getResourceManager()
.RunVdsCommand(
VDSCommandType.CreateSnapshot,
new CreateSnapshotVDSCommandParameters(getStoragePoolId(),
getDestinationStorageDomainId(),
getImageGroupId(),
getImage().getImageId(),
getDiskImage().getSize(),
mNewCreatedDiskImage.getVolumeType(),
mNewCreatedDiskImage.getVolumeFormat(),
getDiskImage().getId(),
getDestinationImageId(),
""));
vdsReturnValue = runVdsCommand(
VDSCommandType.CopyImage,
copyImageParas);
导入导出域中虚拟机
- 使用了 ImportVmCommand→CopyImageGroupCommand→CopyImageVDSCommand,最终执行 CopyImage。
vdcRetValue = Backend.getInstance().runInternalAction(
VdcActionType.CopyImageGroup,
buildMoveOrCopyImageGroupParametersForMemoryConfImage(
containerId, guids.get(0), guids.get(4), guids.get(5)),
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));
vdsReturnValue = runVdsCommand(
VDSCommandType.CopyImage,
copyImageParas);
导入模版
- 使用了 ImportVmTemplateCommand→CopyImageGroupCommand→CopyImageVDSCommand,最终执行 CopyImage。
VdcReturnValueBase vdcRetValue = Backend.getInstance().runInternalAction(
VdcActionType.CopyImageGroup,
p,
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));
vdsReturnValue = runVdsCommand(
VDSCommandType.CopyImage,
copyImageParas);
导出虚拟机
- 使用了 ExportVmCommand→CopyImageGroupCommand→CopyImageVDSCommand,最终执行 CopyImage。
VdcReturnValueBase vdcRetValue = Backend.getInstance().runInternalAction(
VdcActionType.CopyImageGroup,
buildMoveOrCopyImageGroupParametersForMemoryDumpImage(
containerID, guids.get(0), guids.get(2), guids.get(3)),
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));
vdsReturnValue = runVdsCommand(
VDSCommandType.CopyImage,
copyImageParas);
导出模版
- 使用了 ExportVmTemplateCommand→CopyImageGroupCommand→CopyImageVDSCommand,最终执行 CopyImage。
VdcReturnValueBase vdcRetValue = Backend.getInstance().runInternalAction(
VdcActionType.CopyImageGroup,
p,
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));
vdsReturnValue = runVdsCommand(
VDSCommandType.CopyImage,
copyImageParas);
copyImage
uuidReturn = getIrsProxy().copyImage(getParameters().getStorageDomainId().toString(),
getParameters().getStoragePoolId().toString(),
getParameters().getVmId().toString(),
getParameters().getImageGroupId().toString(),
getParameters().getImageId().toString(),
getParameters().getdstImageGroupId().toString(),
getParameters().getDstImageId().toString(),
getParameters().getCopyImageInfo(), // add by hzy for EncryptInfo
getParameters().getImageDescription(),
getParameters().getDstStorageDomainId().toString(),
getParameters().getCopyVolumeType().getValue(),
getParameters().getVolumeFormat().getValue(),
getParameters().getPreallocate().getValue(),
String.valueOf(getParameters().getPostZero()).toLowerCase(),
String.valueOf(getParameters().getForce()).toLowerCase());
public OneUuidReturnForXmlRpc copyImage(String sdUUID, String spUUID, String vmGUID, String srcImgGUID,
String srcVolUUID, String dstImgGUID, String dstVolUUID, Map encryption, String descr, String dstSdUUID, int volType,
int volFormat, int preallocate, String postZero, String force) {
Map xmlRpcReturnValue = irsServer.copyImage(sdUUID, spUUID, vmGUID, srcImgGUID, srcVolUUID,
dstImgGUID, dstVolUUID, encryption, descr, dstSdUUID, volType, volFormat, preallocate, postZero, force);
OneUuidReturnForXmlRpc wrapper = new OneUuidReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
}
- 主机 vdsm,执行 Volume 类的 copy 方法。/usr/share/vdsm/API.py
def copy(self, dstSdUUID, dstImgUUID, dstVolUUID, encryption, desc, volType,
volFormat, preallocate, postZero, force):
vmUUID = '' # vmUUID is never used
return self._irs.copyImage(self._sdUUID, self._spUUID, vmUUID,
self._imgUUID, self._UUID, dstImgUUID,
dstVolUUID,encryption, desc, dstSdUUID, volType,
volFormat, preallocate, postZero, force)
createVolume
uuidReturn = getIrsProxy().createVolume(getParameters().getStorageDomainId().toString(),
getParameters().getStoragePoolId().toString(),
getParameters().getImageGroupId().toString(),
(Long.valueOf(getParameters().getImageSizeInBytes())).toString(),
getParameters().getVolumeFormat().getValue(),
getParameters().getImageType().getValue(),
2,
getParameters().getNewImageID().toString(),
getParameters().getNewImageDescription(),
getParameters().getSourceImageGroupId().toString(),
getParameters().getImageId().toString());
public OneUuidReturnForXmlRpc createVolume(String sdUUID, String spUUID, String imgGUID, String size,
int volFormat, int volType, int diskType, String volUUID, String descr, String srcImgGUID, String srcVolUUID) {
Map xmlRpcReturnValue = irsServer.createVolume(sdUUID, spUUID, imgGUID, size, volFormat,
volType, diskType, volUUID, descr, srcImgGUID, srcVolUUID);
OneUuidReturnForXmlRpc wrapper = new OneUuidReturnForXmlRpc(xmlRpcReturnValue);
return wrapper;
}
- 主机 vdsm,执行 Volume 类的 create 方法。/usr/share/vdsm/API.py
class Volume(APIBase):
ctorArgs = ['volumeID', 'storagepoolID', 'storagedomainID', 'imageID']
class Types:
UNKNOWN = storage.volume.UNKNOWN_VOL
PREALLOCATED = storage.volume.PREALLOCATED_VOL
SPARSE = storage.volume.SPARSE_VOL
class Formats:
UNKNOWN = storage.volume.UNKNOWN_FORMAT
COW = storage.volume.COW_FORMAT
RAW = storage.volume.RAW_FORMAT
class Roles:
SHARED = storage.volume.SHARED_VOL
LEAF = storage.volume.LEAF_VOL
BLANK_UUID = storage.volume.BLANK_UUID
......
def create(self, size, volFormat, preallocate, diskType, desc,
srcImgUUID, srcVolUUID):
return self._irs.createVolume(self._sdUUID, self._spUUID,
self._imgUUID, size, volFormat,
preallocate, diskType, self._UUID, desc,
srcImgUUID, srcVolUUID)
SPM
SPM 是存储池管理器,用于管理所在数据中心的所有存储域,运行在数据中心的其中一台主 机上,通过协调存储域中的元数据来控制对存储的访问。虚拟化平台会保证一直有一个 SPM 在正常运行,在 SPM 的主机出现问题时,会选择另外一台主机。
实现的核心是通过 IrsBrokerCommand抽象类,所有的 irsCommand 都继承该类。调用 vdsm 的接口时,通过 getIrsProxy() 调用。
getIrsProxy 是通过 IrsBrokerCommand 的一个 ConcurrentHashMap,_irsProxyData 的 map 变量,保存了 key 为 storagePool的 id,value 为 IrsProxyData 对象 ,IrsProxyData 保存了与 SPM 主机接口调用的代理 IIrsServer。
private static Map _irsProxyData = new ConcurrentHashMap();
- 一般 IrsBrokerCommand 调用,都是通过 IrsProxyData 找到 IIrsServer 接口 privatemIrsProxy。
private IIrsServer privatemIrsProxy;
private IIrsServer getmIrsProxy() {
return privatemIrsProxy;
}
- gethostFromVds 方法找到合适的主机(状态是 Up,根据优先级,同级的话随机),然后选为 SPM,privatemIrsProxy 为 null,重新选择 SPM。
public IIrsServer getIrsProxy() {
if (getmIrsProxy() == null) {
final StoragePool storagePool = DbFacade.getInstance().getStoragePoolDao().get(_storagePoolId);
// don't try to start spm on uninitialized pool
if (storagePool.getStatus() != StoragePoolStatus.Uninitialized) {
String host =
TransactionSupport.executeInScope(TransactionScopeOption.Suppress,
new TransactionMethod() {
@Override
public String runInTransaction() {
return gethostFromVds();
}
});
if (host != null) {
// Get the values of the timeouts:
int clientTimeOut = Config. getValue(ConfigValues.vdsTimeout) * 1000;
int connectionTimeOut = Config.getValue(ConfigValues.vdsConnectionTimeout) * 1000;
int clientRetries = Config. getValue(ConfigValues.vdsRetries);
Pair returnValue =
XmlRpcUtils.getConnection(host,
getmIrsPort(),
clientTimeOut,
connectionTimeOut,
clientRetries,
IrsServerConnector.class,
Config. getValue(ConfigValues.EncryptHostCommunication));
privatemIrsProxy = new IrsServerWrapper(returnValue.getFirst(), returnValue.getSecond());
runStoragePoolUpEvent(storagePool);
}
}
}
return getmIrsProxy();
}
一般场景下,一台主机挂掉,另一台主机出现 Connecting 状态,是因为调用接口时,出现网络异常 VDSNetworkException。调用 IrsBrokerCommand 的 failover 方法。执行失败后,可尝试恢复两次,恢复次数可设置。
private void failover() {
if ((getParameters().getIgnoreFailoverLimit() || _failoverCounter < Config
. getValue(ConfigValues.SpmCommandFailOverRetries) - 1)
&& getCurrentIrsProxyData().getHasVdssForSpmSelection() && getCurrentIrsProxyData().failover()) {
_failoverCounter++;
executeCommand();
} else {
getVDSReturnValue().setSucceeded(false);
}
}
......
public boolean failover() {
Guid vdsId = mCurrentVdsId;
nullifyInternalProxies();
boolean performFailover = false;
if (vdsId != null) {
try {
VDSReturnValue statusResult = ResourceManager.getInstance().runVdsCommand(VDSCommandType.SpmStatus,
new SpmStatusVDSCommandParameters(vdsId, _storagePoolId));
if (statusResult != null
&& statusResult.getSucceeded()
&& (((SpmStatusResult) statusResult.getReturnValue()).getSpmStatus() == SpmStatus.SPM || ((SpmStatusResult) statusResult
.getReturnValue()).getSpmStatus() == SpmStatus.Contend)) {
performFailover = ResourceManager
.getInstance()
.runVdsCommand(VDSCommandType.SpmStop,
new SpmStopVDSCommandParameters(vdsId, _storagePoolId)).getSucceeded();
} else {
performFailover = true;
}
} catch (Exception ex) {
// try to failover to another host if failed to get spm
// status or stop spm
// (in case mCurrentVdsId has wrong id for some reason)
log.errorFormat("Could not get spm status on host {0} for spmStop.", vdsId);
performFailover = true;
}
}
if (performFailover) {
log.infoFormat("Irs placed on server {0} failed. Proceed Failover", vdsId);
mTriedVdssList.add(vdsId);
return true;
} else {
log.errorFormat("IRS failover failed - cant allocate vds server");
return false;
}
}