The Universal Service Manager (USM)
Overview
The Universal Service Manager (USM) is a GigaSpaces processing unit specially designed to install, execute, monitor and manage an external process. The USM manages the lifecycle of the external process, and acts as a proxy for it in the GigaSpaces Service Grid. This allows operators to deploy any application to the cloud via the service grid, and gain all of its benefits (Centralized management, remote deployment, performance statistics, etc…)
How it works
Setting up a server process usually means setting up the host (physical or virtual), downloading and unzipping the binaries, setting up configuration files and scripts for start-up/shutdown. The USM helps you automate this process on a machine running in the GigaSpaces service grid. With the USM, you set up a file with all of the details required to download, install, run and monitor your process. This file, called a recipe, uses a Domain Specific Language (DSL) specifically designed to describe the process of automating process deployment and monitoring. The USM project also offers built-in plugins and tools which can be used for common scenarios. The Recipe DSL makes it easy to describe how your process should be installed and executed, using either command line operations (including windows batch files, Unix shell scripts and OS-independent groovy scripts) or small snippets of code called closure which can be embedded into the recipe file. (More information is available online for Groovy and Closures)
Ideally, the process you are running already comes with the required scripts to start and stop it, so a simple recipe will only require you to set the names of these scripts, and you’re ready to go.
What is a recipe
A recipe is basically a list of instructions that the USM will execute when required. It is intended as a tool that can be used by someone with little or no development experience, but with a background in operations. More details are available in Developing Recipes.
USM Lifecycle
The life-cycle phases of the USM can be controlled by the user, allowing for a high level of customization. TheUSM life-cycle is split into phases, each contains one or more events. The recipe can implement any of these events and ignore the ones it does not need. The only mandatory event is the ‘start’ event, which starts up the external process. The life-cycle phases and events are described below.
Service Start Phase
events: preServiceStart
Called once when a service starts. This event will only execute on the first instance of the service that launches (which has an instance ID of 1).
Note that other instances may be launching at the same time, so their init Phase, may be executing at the same time at a different instance.
Initialization Phase
events: init
Called when the USM first starts up. Should be used to validate system level settings (supported operating system, hardware architecture, etc…)
Install Phase
events: preInstall, install, postInstall
Called after the init phase ends. This is where an application is expected to setup any binaries it requires to run.
Note that a future version of Cloudify will provide built-in integration with standard package managers, like apt, yum, msi or maven.
Start Phase:
events: preStart, start, startDetection, postStart
The start event is the only mandatory event, where the external process is launched. The events should be used as following:
- preStart – check that required operating system files are available, like files, disk space and ports.
- start – Launches the external process. This event has specific requirements (see below)
- startDetection – After the USM launches the external process in the start event, the application should notify it when the external process is considered ‘ready’. In practice, the USM often launches
an operating system shell for a start up script which then launches the actual process. The USM does not necessarily know when the actual process is ready, to this event offers a hint. When this event finishes,
the USM will scan the operating system for the process tree it just launched (i.e. the process launched in the start event, and all of its children
Stop Phase:
events: preStop, postStop
By default, the USM ‘stops’ the external process by killing the process using an operating system call (using the Sigar library). This is an effective approach
for most cases, but may not work for some cases where a process needs to perform specific operations during shutdown, like persisting state to disk. In these cases,
the recipe should use the preStop event to handle any additional logic. For instance, a tomcat recipe may call the shutdown script on the preStop phase, then wait
until the shutdown command closes the tomcat process.
Shutdown Phase:
events: shutdown
The shutdown event should be used to perform any required cleanup before the USM instance shuts down. It is often used when running on physical hosts
to delete any files that were saved to a location on the local file system that is not under the default service directory (which is deleted automatically when
the USM instance is un-deployed).
Service Installation
The service recipe includes an install phase, where a recipe is expected to set up the files it requires to execute correctly.
The files are typically made available using one of the following ways, or a combination of them:
- Download the file from the internet, using a utility like wget or a package repository like yum, apt or maven.
- Download the file from a local network resource, like a network share.
- Assume the files are available on the file system of the host.
- Include the required files in the packaged service folder, either as a compressed archive that will be uncompressed during the install phase, or as the uncompressed
files.
Each of these options has an impact on the architecture of the deployed system:
- Downloading the files from the internet requires an outgoing connection to the internet (which may not be available in some environments) as well as sufficient
bandwidth to ensure a speedy download. The time required to download the files is also a factor, as a slow download will significantly increase the start-up time
of each of the instances. Finally, the source site MUST be available at all times, in case a host or a service instance fails and a USM instance has to be
provisioned. Cloud Blob Stores are often a good candidate for a source site. - When downloading from a local network resource, the resource must be highly available. A single NFSserver, for instance, is a bad choice, as a failure of the
NFS server would make recovery of a USM instance impossible. - This option usually implies using a Virtual Machine that has the required binaries pre-installed, or using a fixed set of machines that have been previously
set up with the required files. This approach is a good fit for applications that expect to be deployed on a fairly static well-known system, or on one specific
cloud vendor. - Packaging the process files inside the PU will make the packaged file considerably larger.
The current version of the GigaSpaces Service Grid should not be used with files larger then 150 MBVERIFY THIS, but for smaller sizes, the packaged PU is kept on at least two hosts (the hosts running the GigaSpaces Grid Service Managers). A future version will support larger files.
Executable Recipe Entries
Several of the recipe parameters are executable entries – entries that define execution of recipe-specific code, like executing a shell script.
The USM supports several ways to define an executable entry. Each of these way has a different input.
- Command line (String): The string indicates a command line that should be executed as an external process. If spaces exist in the command line, each of the words is considered a single parameter and a String Array is created from all of the words, so that the first word is the executable to run, and the others are parameters to the executable.
- A list: Similar to the command line option, this option defines the command line by stating its parts explicitly, allowing a user to pass parameters that contains spaces.
Note that handling of spaces, single quotes and double quotes is operating system specific, and it is the responsibility of the recipe developer to handle this correctly.
- A closure: An executable piece of code that is executed inside the USM’s JVM. This option offers the best performance and allows you to avoid additional files in your recipe, but requires a working knowledge of the Groovy programming language. In addition, it is important to note that the working folder of the code executed in a closure is the working folder of the USM’s JVM (i.e. the GigaSpaces Grid Services Container). This is usually the bin directory under the GigaSpaces install directory, and not the USM’s service folder. To find the service folder, use the Service Context.
- A map: In some cases, a recipe is expected to run on multiple operating systems. Each operating system may require a slightly different implementation. The map option allows you to define a mapping for each operating system. The map’s key is a string that represents a Java Regular Expression (See more details in the “Javadoc”/http://download.oracle.com/javase/6/docs/api/index.html?java/util/regex/package-summary.html and its value should be one of the previously described options.
The USM will attempt to match the operating system it is running on (using Java’s ‘os.name’ attribute) to the keys in the map, and execute the first entry that is a match. Typical keys are ‘Linux’, ‘Win.\*’, ‘Mac.\*’ (all recent apple operating systems).
A list of operating system names is available “here”.http://lopica.sourceforge.net/os.html.
All external processes are launched using the service directory (‘ext’ in the Processing Unit) as the process’ working directory.
Command line handling
The USM performs some common modifications to the given command line, based on some common use cases:
- On windows, if the first word in the command line ends with ‘.bat’, indicating that it is a windows batch file, the command is prepended with ‘cmd.exe’ and ‘/c’. This is the correct way to execute a batch file.
ndows machines, if the first parameter is a file that in the recipe service folder or one of its subdirectories (to any level) the file is marked as executable before the command line is launched. - If the first word in the command line ends with ‘.groovy’, the USM will launch the groovy processor provided with GigaSpaces (in tools/groovy under the GigaSpaces installation directory) passing the given command line to it. It will also set the CLASSPATH environment variable, making specific jar files available to the Groovy runtime, like the files in the PU’s lib dir.
Complex command lines
The USM is not a replacement for the operating system’s command shell.
It does not handle parameters, single and double quotes, white spaces and other special characters the same way that the shell does.
If your command line is complex, it is recommended that you copy the command to a script file (windows batch file, unix shell script, etc) and use the new script file’s name as the command line.
This will ensure that your command is executed the way it was meant to, just like it was executed from a shell.
Plugins
The recipe file is intended to handle most of the common use cases, but it is impossible to handle all scenarios.
Eventually, you may encounter a scenario that requires writing custom code to handle a certain kind of event or performance statistic.
This is what the plugin mechanism allows. A plugin is a java class that implements theorg.cloudifysource.dsl.Plugin interface, plus one or more additional interfaces as described here.
The POJO class name, plus a map of parameters,is defined in the recipe file.
When the USM loads, it will execute the plugin according to the interfaces it implements, giving the developer the required flexibility.
The jar file containing the compiled plugin implementation, plus any additional classes and resources it needs, should be places in your recipe folder before it is packaged, under the usmlib directory.
When the PU is deployed, these files will be placed in the PU’s lib dir, making them available to the USM’s classloader.
How plugins works
A class implementing a plugin should implement the interfaces that it would like to hook into (like specific events, monitors, etc) and it must have a default constructor.
When the USM starts up, it reads the DSL file and creates an instance of the class according to the classname specified in the recipe file, and passes to this object the parameters map that is also specified in the recipe file.
This object is then added to the USM’s Spring Application Context, making it a singleton spring bean. The object will be injected to the various hooks in the USM according to the interfaces that the plugin implements.
Monitors and Details
The GigaSpaces Service Grid allows a Processing Unit instance to publish data about itself in a key/value format, for both static data (ServiceDetails) which remains constant for the lifetime of the PU, and dynamic data(ServiceMonitors) which changes all the time.
The USM publishes a set of details and monitors, mostly related to the operating system statistics of the monitored process, but A Recipe may add additional details and monitors if required.
At this time, adding monitors or details requires implementing a DSL plugin though a future update will allow a recipe developer to define the details/monitors directly in the recipe.
To publish monitors or details, the plugin should implement one or borh of the relevant interfaces:
org.openspaces.usm.details.Details and org.openspaces.usm.monitors.Monitor.
USM Technical Design
This section describes some of the technical details of the implementation of the USM implementation. Familiarity with these details is not required to
create most recipe, but is useful for the more advanced implementations.
Implementing events and other integration points
The USM relies heavily on spring based dependency injection. All user-controlled events and integration points are implemented as POJOs that implement one or more of the USM interfaces. For example, the ‘init’ event is handled by a POJO that implements the org.openspaces.usm.events.InitListener interface. During the USM start-up, it loads the DSL file and uses the settings defined there to create a set of POJOs which implements the behavior described in the recipe.
This diagram shows all of the available interfaces:
Processing Unit Structure
The USM is deployed just like a standard XAP Processing Unit.
It includes a standard pu.xml file, the usm.jar implementation files (available in the XAP installation folder under lib/platform/usm) and an additional folder, called ‘ext’, where the service recipe and any additional resources are placed.
PU
+ lib
\| usm.jar
\|
+ META-INF
+ spring
| pu.xml
+ ext
| -service.groovy
|
Process Handling
The USM acts as a proxy to an external process defined by the recipe, representing this process in the GigaSpaces service grid. To do this, it monitors Operating System metrics for this process, like CPU Usage, Memory consumption, etc. But to be able to do this, the USM must first find the process. This sounds easy, since the USM is the component that launches the external script.
But in reality, the USM may launch a process that launches another process, that launches another one, etc. For instance, a process may be launched from a groovy script. This translates to the USM launching a shell process like bash for the groovy launch script, which launches a JVM that runs the Groovy Script, which launches the actual process. It is this final process that the USM should monitor, not the initial shell process that it launched.
The USM handles this by comparing the operating system’s process tree before the external process is launched, and after it was launched. A new process whose parent process is the USM’s JVM, is called the child process. The USM then looks for a process that was launched by the child process in a similar fashion, and continues down the process tree until a ‘leaf’ process is found, that has no child processes. This process is called theactual process, and this is the process that the USM will monitor for performance statistics.
The child process and the actual process may be the same.
Before the USM can find the actual process, it must first know that the actual process has launched! Launching the chain of processes that eventually starts the actual process may take an indeterminate amount of time. It is for this reason that the USM supports the start Detection event, which is how the recipe determines when the actual process has started. This is most often accomplished by waiting for a port to open on the local host (like port 80 for a web server) or for a certain text message to be written to a log file. For example, by polling the port until it becomes available, the recipe can indicate to the USM when the actual process is ready. A utility for scanning a port is packaged with the USM, though the recipe can configure any arbitrary code to determine this.