Configuring and Managing Cluster Resources (Command Line)

Chapter 7. Configuring and Managing Cluster Resources (Command Line)¶

Abstract

To configure and manage cluster resources, either use the graphical user interface (the Pacemaker GUI) or the crm command line utility. For the GUI approach, refer to Chapter 5, Configuring and Managing Cluster Resources (GUI).

This chapter introduces crm, the command line tool and covers an overview of this tool, how to use templates, and mainly configuring and managing cluster resources: creating basic and advanced types of resources (groups and clones), configuring constraints, specifying failover nodes and failback nodes, configuring resource monitoring, starting, cleaning up or removing resources, and migrating resources manually.

User Privileges

Sufficient privileges are necessary to manage a cluster. The crm command and its subcommands have to be run either as root user or as the CRM owner user (typically the user hacluster). However, the user option allows you to run crm and its subcommands as a regular (unprivileged) user and to change its ID using sudo whenever necessary. For example, with the following command crm will use hacluster as the privileged user ID: crm options user hacluster Note that you need to set up /etc/sudoers so that sudo does not ask for a password.

7.1. crm Shell—Overview¶

After installation you usually need the crm command only. This command has several subcommands which manage resources, CIBs, nodes, resource agents, and others. Run crm help to get an overview of all available commands. It offers a thorough help system with embedded examples.

The crm command can be used in the following ways:

• Directly.  Concatenate all subcommands to crm, press Enter and you see the output immediately. For example, enter crm help ra to get information about the ra subcommand (resource agents).

• As crm Shell Script.  Use crm and its commands in a script. This can be done in two ways:

  crm -f script.cli
  crm < script.cli

  The script can contain any command from crm. For example:

  # A small example
  statusnode list

  Any line starting with the hash symbol (#) is a comment and is ignored. If a line is too long, insert a backslash (\) at the end and continue in the next line.

• Interactive as Internal Shell.  Type crm to enter the internal shell. The prompt changes to crm(live)#. With help you can get an overview of the available subcommands. As the internal shell has different levels of subcommands, you can “enter” one by just typing this subcommand and press Enter.

  For example, if you type resource you enter the resource management level. Your prompt changes to crm(live)resource#. If you want to leave the internal shell, use the commands quit, bye, or exit. If you need to go one level back, use up, end, or cd.

  You can enter the level directly by typing crm and the respective subcommand(s) without any options and hit Enter.

  The internal shell supports also tab completion for subcommands and resources. Type the beginning of a command, press →| and crm completes the respective object.

In addition to previously explained methods, the crm shell also supports synchronous command execution. Use the -w option to activate it. If you have started crm without -w, you can enable it later with the user preference's wait set to yes (options wait yes). If this option is enabled, crm waits until the transition is finished. Whenever a transaction is started, dots are printed to indicate progress. Synchronous command execution is only applicable for commands like resource start.

	Differentiate Between Management and Configuration Subcommands
The crm tool has management capability (the subcommands resource and node) and can be used for configuration (cib, configure).

The following subsections give you an overview about some important aspects of the crm tool.

7.1.1. Displaying Information about OCF Resource Agents¶

As you have to deal with resource agents in your cluster configuration all the time, the crm tool contains the ra command to get information about resource agents and to manage them (for additional information, see also Section 4.2.2, “Supported Resource Agent Classes”):

# crm ra
crm(live)ra#

The command classes gives you a list of all classes and providers:

crm(live)ra# classes
heartbeat
lsb
ocf / heartbeat linbit lvm2 ocfs2 pacemaker
stonith

To get an overview about all available resource agents for a class (and provider) use the list command:

crm(live)ra# list ocf
AoEtarget           AudibleAlarm        CTDB                ClusterMon
Delay               Dummy               EvmsSCC             Evmsd
Filesystem          HealthCPU           HealthSMART         ICP
IPaddr              IPaddr2             IPsrcaddr           IPv6addr
LVM                 LinuxSCSI           MailTo              ManageRAID
ManageVE            Pure-FTPd           Raid1               Route
SAPDatabase         SAPInstance         SendArp             ServeRAID
...

An overview about a resource agent can be viewed with info:

crm(live)ra# info ocf:drbd:linbit
This resource agent manages a DRBD* resource
as a master/slave resource. DRBD is a shared-nothing replicated storage
device. (ocf:linbit:drbd)

Master/Slave OCF Resource Agent for DRBD

Parameters (* denotes required, [] the default):

drbd_resource* (string): drbd resource name
    The name of the drbd resource from the drbd.conf file.

drbdconf (string, [/etc/drbd.conf]): Path to drbd.conf
    Full path to the drbd.conf file.

Operations' defaults (advisory minimum):

    start         timeout=240
    promote       timeout=90 
    demote        timeout=90 
    notify        timeout=90 
    stop          timeout=100
    monitor_Slave_0 interval=20 timeout=20 start-delay=1m
    monitor_Master_0 interval=10 timeout=20 start-delay=1m

Leave the viewer by pressing Q. Find a configuration example at Appendix A, Example of Setting Up a Simple Testing Resource.

	Use crm Directly
In the former example we used the internal shell of the crm command. However, you do not necessarily have to use it. You get the same results, if you add the respective subcommands to crm. For example, you can list all the OCF resource agents by entering crm ra list ocf in your shell.

7.1.2. Using Configuration Templates¶

Configuration templates are ready-made cluster configurations for the crm shell. Do not confuse them with the resource templates (as described in Section 7.3.2, “Creating Resource Templates”). Those are templates for the cluster and not for the crm shell.

Configuration templates require minimum effort to be tailored to the particular user's needs. Whenever a template creates a configuration, warning messages give hints which can be edited later for further customization.

The following procedure shows how to create a simple yet functional Apache configuration:

1. Log in as root and start the crm tool:

   # crm configure

2. Create a new configuration from a configuration template:

   1. Switch to the template subcommand:

      crm(live)configure# template

   2. List the available configuration templates:

      crm(live)configure template#  list templates
      gfs2-base   filesystem  virtual-ip  apache   clvm     ocfs2    gfs2

   3. Decide which configuration template you need. As we need an Apache configuration, we choose the apache template:

      crm(live)configure template#  new intranet apache
      INFO: pulling in template apache
      INFO: pulling in template virtual-ip

3. Define your parameters:

   1. List the just created configuration:

      crm(live)configure template#  list
      intranet

   2. Display the minimum of required changes which have to be filled out by you:

      crm(live)configure template#  show
      ERROR: 23: required parameter ip not set
      ERROR: 61: required parameter id not set
      ERROR: 65: required parameter configfile not set

   3. Invoke your preferred text editor and fill out all lines that have been displayed as errors in Step 3.b:

      crm(live)configure template#  edit

4. Show the configuration and check whether it is valid (bold text depends on the configuration you have entered in Step 3.c):

   crm(live)configure template#  show
   primitive virtual-ip ocf:heartbeat:IPaddr \
       params ip="192.168.1.101"
   primitive apache ocf:heartbeat:apache \
       params configfile="/etc/apache2/httpd.conf"
   monitor apache 120s:60s
   group intranet \
       apache virtual-ip

5. Apply the configuration:

   crm(live)configure template#  apply
   crm(live)configure#  cd ..
   crm(live)configure#  show

6. Submit your changes to the CIB:

   crm(live)configure#  commit

It is possible to simplify the commands even more, if you know the details. The above procedure can be summarized with the following command on the shell:

crm configure template \
   new intranet apache params \
   configfile="/etc/apache2/httpd.conf" ip="192.168.1.101"

If you are inside your internal crm shell, use the following command:

crm(live)configure template# new intranet apache params \
   configfile="/etc/apache2/httpd.conf" ip="192.168.1.101"

However, the previous command only creates its configuration from the configuration template. It does not apply nor commit it to the CIB.

7.1.3. Testing with Shadow Configuration¶

A shadow configuration is used to test different configuration scenarios. If you have created several shadow configurations, you can test them one by one to see the effects of your changes.

The usual process looks like this:

1. Log in as root and start the crm tool:

   # crm configure

2. Create a new shadow configuration:

   crm(live)configure# cib new myNewConfig
   INFO: myNewConfig shadow CIB created

3. If you want to copy the current live configuration into your shadow configuration, use the following command, otherwise skip this step:

   crm(myNewConfig)# cib reset myNewConfig

   The previous command makes it easier to modify any existing resources later.

4. Make your changes as usual. After you have created the shadow configuration, all changes go there. To save all your changes, use the following command:

   crm(myNewConfig)# commit

5. If you need the live cluster configuration again, switch back with the following command:

   crm(myNewConfig)configure# cib use live
   crm(live)#

7.1.4. Debugging Your Configuration Changes¶

Before loading your configuration changes back into the cluster, it is recommended to review your changes with ptest. The ptest can show a diagram of actions that will be induced by committing the changes. You need the graphviz package to display the diagrams. The following example is a transcript, adding a monitor operation:

# crm configure
crm(live)configure# show fence-node2 
primitive fence-node2 stonith:apcsmart \
        params hostlist="node2"
crm(live)configure# monitor fence-node2 120m:60s
crm(live)configure# show changed
primitive fence-node2 stonith:apcsmart \
        params hostlist="node2" \
        op monitor interval="120m" timeout="60s"
crm(live)configure# ptest
crm(live)configure# commit

7.2. Configuring Global Cluster Options¶

Global cluster options control how the cluster behaves when confronted with certain situations. The predefined values can be kept in most cases. However, to make key functions of your cluster work correctly, you need to adjust the following parameters after basic cluster setup:

Procedure 7.1. Modifying Global Cluster Options With crm

1. Log in as root and start the crm tool:

   # crm configure

2. Use the following commands to set the options for a two-node clusters only:

   crm(live)configure# property no-quorum-policy=ignore
   crm(live)configure# property stonith-enabled=false

3. Show your changes:

   crm(live)configure# show
   property $id="cib-bootstrap-options" \
      dc-version="1.1.1-530add2a3721a0ecccb24660a97dbfdaa3e68f51" \
      cluster-infrastructure="openais" \
      expected-quorum-votes="2" \
      no-quorum-policy="ignore" \
      stonith-enabled="false"

4. Commit your changes and exit:

   crm(live)configure# commit
   crm(live)configure# exit

7.3. Configuring Cluster Resources¶

As a cluster administrator, you need to create cluster resources for every resource or application you run on servers in your cluster. Cluster resources can include Web sites, e-mail servers, databases, file systems, virtual machines, and any other server-based applications or services you want to make available to users at all times.

For an overview of resource types you can create, refer to Section 4.2.3, “Types of Resources”.

7.3.1. Creating Cluster Resources¶

There are three types of RAs (Resource Agents) available with the cluster (for background information, see Section 4.2.2, “Supported Resource Agent Classes”). To create a cluster resource use the crm tool. To add a new resource to the cluster, proceed as follows:

1. Log in as root and start the crm tool:

   # crm configure

2. Configure a primitive IP address:

   crm(live)configure# primitive myIP ocf:heartbeat:IPaddr \
        params ip=127.0.0.99 op monitor interval=60s

   The previous command configures a “primitive” with the name myIP. You need to choose a class (here ocf), provider (heartbeat), and type (IPaddr). Furthermore, this primitive expects other parameters like the IP address. Change the address to your setup.

3. Display and review the changes you have made:

   crm(live)configure# show

4. Commit your changes to take effect:

   crm(live)configure# commit

7.3.2. Creating Resource Templates¶

If you want to create several resources with similar configurations, a resource template simplifies the task. See also Section 4.4.3, “Resource Templates and Constraints” for some basic background information. Do not confuse them with the “normal” templates from Section 7.1.2, “Using Configuration Templates”. Use the rsc_template command to get familiar with the syntax:

# crm configure rsc_template
usage: rsc_template <name> [<class>:[<provider>:]]<type>
        [params <param>=<value> [<param>=<value>...]]
        [meta <attribute>=<value> [<attribute>=<value>...]]
        [utilization <attribute>=<value> [<attribute>=<value>...]]
        [operations id_spec
            [op op_type [<attribute>=<value>...] ...]]

For example, the following command creates a new resource template with the name BigVM derived from the ocf:heartbeat:Xen resource and some default values and operations:

crm(live)configure# rsc_template BigVM ocf:heartbeat:Xen \
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

Once you defined the new resource template, you can use it in primitives or reference it in order, colocation, or rsc_ticket constraints. To reference the resource template, use the @ sign:

crm(live)configure# primitive MyVM1 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1"

The new primitive MyVM1 is going to inherit everything from the BigVM resource templates. For example, the equivalent of the above two would be:

crm(live)configure# primitive MyVM1 ocf:heartbeat:Xen \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1"
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

If you want to overwrite some options or operations, add them to your (primitive) definition. For instance, the following new primitive MyVM2 doubles the timout for monitor operations but leaves others untouched:

crm(live)configure# primitive MyVM2 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM2" name="MyVM2" \
   op monitor timeout=120s interval=30s    

A resource template may be referenced in constraints to stand for all primitives which are derived from that template. This helps to produce a more concise and clear cluster configuration. Resource template references are allowed in all constraints except location constraints. Colocation constraints may not contain more than one template reference.

7.3.3. Creating a STONITH Resource¶

From the crm perspective, a STONITH device is just another resource. To create a STONITH resource, proceed as follows:

1. Log in as root and start the crm tool:

   # crm configure

2. Get a list of all STONITH types with the following command:

   crm(live)# ra list stonith
   apcmaster                  apcmastersnmp              apcsmart
   baytech                    bladehpi                   cyclades
   drac3                      external/drac5             external/dracmc-telnet
   external/hetzner           external/hmchttp           external/ibmrsa
   external/ibmrsa-telnet     external/ipmi              external/ippower9258
   external/kdumpcheck        external/libvirt           external/nut
   external/rackpdu           external/riloe             external/sbd
   external/vcenter           external/vmware            external/xen0
   external/xen0-ha           fence_legacy               ibmhmc
   ipmilan                    meatware                   nw_rpc100s
   rcd_serial                 rps10                      suicide
   wti_mpc                    wti_nps

3. Choose a STONITH type from the above list and view the list of possible options. Use the following command:

   crm(live)# ra info stonith:external/ipmi
   IPMI STONITH external device (stonith:external/ipmi)
   
   ipmitool based power management. Apparently, the power off
   method of ipmitool is intercepted by ACPI which then makes
   a regular shutdown. If case of a split brain on a two-node
   it may happen that no node survives. For two-node clusters
   use only the reset method.
   
   Parameters (* denotes required, [] the default):
   
   hostname (string): Hostname
       The name of the host to be managed by this STONITH device.
   ...

4. Create the STONITH resource with the stonith class, the type you have chosen in Step 3, and the respective parameters if needed, for example:

   crm(live)# configure
   crm(live)configure# primitive my-stonith stonith:external/ipmi \
       params hostname="node1"
       ipaddr="192.168.1.221" \
       userid="admin" passwd="secret" \
       op monitor interval=60m timeout=120s  

7.3.4. Configuring Resource Constraints¶

Having all the resources configured is only one part of the job. Even if the cluster knows all needed resources, it might still not be able to handle them correctly. For example, try not to mount the file system on the slave node of drbd (in fact, this would fail with drbd). Define constraints to make these kind of information available to the cluster.

For more information about constraints, see Section 4.4, “Resource Constraints”.

7.3.4.1. Locational Constraints¶

This type of constraint may be added multiple times for each resource. All location constraints are evaluated for a given resource. A simple example that expresses a preference to run the resource fs1 on the node with the name earth to 100 would be the following:

crm(live)configure# location fs1-loc fs1 100: earth

Another example is a location with pingd:

crm(live)configure# primitive pingd pingd \
    params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
crm(live)configure#  location node_pref internal_www \
    rule 50: #uname eq node1 \
    rule pingd: defined pingd

7.3.4.2. Colocational Constraints¶

The colocation command is used to define what resources should run on the same or on different hosts.

It is only possible to set a score of either +inf or -inf, defining resources that must always or must never run on the same node. It is also possible to use non-infinite scores. In that case the colocation is called advisory and the cluster may decide not to follow them in favor of not stopping other resources if there is a conflict.

For example, to run the resources with the IDs filesystem_resource and nfs_group always on the same host, use the following constraint:

crm(live)configure# colocation nfs_on_filesystem inf: nfs_group filesystem_resource

For a master slave configuration, it is necessary to know if the current node is a master in addition to running the resource locally.

7.3.4.3. Ordering Constraints¶

Sometimes it is necessary to provide an order of resource actions or operations. For example, you cannot mount a file system before the device is available to a system. Ordering constraints can be used to start or stop a service right before or after a different resource meets a special condition, such as being started, stopped, or promoted to master. Use the following commands in the crm shell to configure an ordering constraint:

crm(live)configure# order nfs_after_filesystem mandatory: filesystem_resource nfs_group

7.3.4.4. Constraints for the Example Configuration¶

The example used for this chapter would not work without additional constraints. It is essential that all resources run on the same machine as the master of the drbd resource. The drbd resource must be master before any other resource starts. Trying to mount the DRBD device when it is not the master simply fails. The following constraints must be fulfilled:

• The file system must always be on the same node as the master of the DRBD resource.

  crm(live)configure# colocation filesystem_on_master inf: \
      filesystem_resource drbd_resource:Master

• The NFS server as well as the IP address must be on the same node as the file system.

  crm(live)configure# colocation nfs_with_fs inf: \
     nfs_group filesystem_resource

• The NFS server as well as the IP address start after the file system is mounted:

  crm(live)configure# order nfs_second mandatory: \
     filesystem_resource:start nfs_group

• The file system must be mounted on a node after the DRBD resource is promoted to master on this node.

  crm(live)configure# order drbd_first inf: \
      drbd_resource:promote filesystem_resource:start

7.3.5. Specifying Resource Failover Nodes¶

To determine a resource failover, use the meta attribute migration-threshold. In case failcount exceeds migration-threshold on all nodes, the resource will remain stopped. For example:

crm(live)configure# location r1-node1 r1 100: node1

Normally, r1 prefers to run on node1. If it fails there, migration-threshold is checked and compared to the failcount. If failcount >= migration-threshold then it is migrated to the node with the next best preference.

Start failures set the failcount to inf depend on the start-failure-is-fatal option. Stop failures cause fencing. If there is no STONITH defined, the resource will not migrate at all.

For an overview, refer to Section 4.4.4, “Failover Nodes”.

7.3.6. Specifying Resource Failback Nodes (Resource Stickiness)¶

A resource might fail back to its original node when that node is back online and in the cluster. If you want to prevent a resource from failing back to the node it was running on prior to failover, or if you want to specify a different node for the resource to fail back to, you must change its resource stickiness value. You can either specify resource stickiness when you are creating a resource, or afterwards.

For an overview, refer to Section 4.4.5, “Failback Nodes”.

7.3.7. Configuring Placement of Resources Based on Load Impact¶

Some resources may have specific capacity requirements such as minimum amount of memory. Otherwise, they may fail to start completely or run with degraded performance.

To take this into account, the High Availability Extension allows you to specify the following parameters:

1. The capacity a certain node provides.

2. The capacity a certain resource requires.

3. An overall strategy for placement of resources.

For detailed background information about the parameters and a configuration example, refer to Section 4.4.6, “Placing Resources Based on Their Load Impact”.

To configure the resource's requirements and the capacity a node provides, use utilization attributes as described in Procedure 5.10, “Adding Or Modifying Utilization Attributes”. You can name the utilization attributes according to your preferences and define as many name/value pairs as your configuration needs.

In the following example, we assume that you already have a basic configuration of cluster nodes and resources and now additionally want to configure the capacities a certain node provides and the capacity a certain resource requires.

Procedure 7.2. Adding Or Modifying Utilization Attributes With crm

1. Log in as root and start the crm tool:

   # crm configure

2. To specify the capacity a node provides, use the following command and replace the placeholder NODE_1 with the name of your node:

   crm(live)configure# node
   NODE_1 utilization memory=16384 cpu=8

   With these values, NODE_1 would be assumed to provide 16GB of memory and 8 CPU cores to resources.

3. To specify the capacity a resource requires, use:

   crm(live)configure# primitive
    xen1 ocf:heartbeat:Xen ... \
        utilization memory=4096 cpu=4

   This would make the resource consume 4096 of those memory units from nodeA, and 4 of the cpu units.

4. Configure the placement strategy with the property command:

   crm(live)configure# property ...

   Four values are available for the placement strategy:

   propertyplacement-strategy=default

   Utilization values are not taken into account at all, per default. Resources are allocated according to location scoring. If scores are equal, resources are evenly distributed across nodes.

   propertyplacement-strategy=utilization

   Utilization values are taken into account when deciding whether a node is considered eligible if it has sufficient free capacity to satisfy the resource's requirements. However, load-balancing is still done based on the number of resources allocated to a node.

   propertyplacement-strategy=minimal

   Utilization values are taken into account when deciding whether a node is eligible to serve a resource; an attempt is made to concentrate the resources on as few nodes as possible, thereby enabling possible power savings on the remaining nodes.

   propertyplacement-strategy=balanced

   Utilization values are taken into account when deciding whether a node is eligible to serve a resource; an attempt is made to spread the resources evenly, optimizing resource performance.

   The placing strategies are best-effort, and do not yet utilize complex heuristic solvers to always reach an optimum allocation result. Ensure that resource priorities are properly set so that your most important resources are scheduled first.

5. Commit your changes before leaving the crm shell:

   crm(live)configure# commit

The following example demonstrates a three node cluster of equal nodes, with 4 virtual machines:

crm(live)configure# node node1 utilization memory="4000"
crm(live)configure# node node2 utilization memory="4000"
crm(live)configure# node node3 utilization memory="4000"
crm(live)configure# primitive xenA ocf:heartbeat:Xen \
    utilization memory="3500" meta priority="10"
crm(live)configure# primitive xenB ocf:heartbeat:Xen \
    utilization memory="2000" meta priority="1"
crm(live)configure# primitive xenC ocf:heartbeat:Xen \
    utilization memory="2000" meta priority="1"
crm(live)configure# primitive xenD ocf:heartbeat:Xen \
    utilization memory="1000" meta priority="5"
crm(live)configure# property placement-strategy="minimal"

With all three nodes up, xenA will be placed onto a node first, followed by xenD. xenB and xenC would either be allocated together or one of them with xenD.

If one node failed, too little total memory would be available to host them all. xenA would be ensured to be allocated, as would xenD; however, only one of xenB or xenC could still be placed, and since their priority is equal, the result is not defined yet. To resolve this ambiguity as well, you would need to set a higher priority for either one.

7.3.8. Configuring Resource Monitoring¶

To monitor a resource, there are two possibilities: either define a monitor operation with the op keyword or use the monitor command. The following example configures an Apache resource and monitors it every 60 seconds with the op keyword:

crm(live)configure# primitive apache apache \
  params ... \
  op monitor interval=60s timeout=30s

The same can be done with:

crm(live)configure# primitive apache apache \
   params ...
crm(live)configure# monitor apache 60s:30s

For an overview, refer to Section 4.3, “Resource Monitoring”.

7.3.9. Configuring a Cluster Resource Group¶

One of the most common elements of a cluster is a set of resources that needs to be located together. Start sequentially and stop in the reverse order. To simplify this configuration we support the concept of groups. The following example creates two primitives (an IP address and an e-mail resource):

1. Run the crm command as system administrator. The prompt changes to crm(live).

2. Configure the primitives:

   crm(live)# configure
   crm(live)configure# primitive Public-IP ocf:IPaddr:heartbeat \
      params ip=1.2.3.4
   crm(live)configure# primitive Email lsb:exim

3. Group the primitives with their relevant identifiers in the correct order:

   crm(live)configure# group shortcut Public-IP Email

For an overview, refer to Section 4.2.5.1, “Groups”.

7.3.10. Configuring a Clone Resource¶

Clones were initially conceived as a convenient way to start N instances of an IP resource and have them distributed throughout the cluster for load balancing. They have turned out to quite useful for a number of other purposes, including integrating with DLM, the fencing subsystem and OCFS2. You can clone any resource, provided the resource agent supports it.

Learn more about cloned resources in Section 4.2.5.2, “Clones”.

7.3.10.1. Creating Anonymous Clone Resources¶

To create an anonymous clone resource, first create a primitive resource and then refer to it with the clone command. Do the following:

1. Log in as root and start the crm tool:

   # crm configure

2. Configure the primitive, for example:

   crm(live)configure# primitive Apache lsb:apache

3. Clone the primitive:

   crm(live)configure# clone apache-clone Apache 

7.3.10.2. Creating Stateful/Multi-State Clone Resources¶

To create an stateful clone resource, first create a primitive resource and then the master/slave resource. The master/slave resource must support at least promote and demote operations.

1. Log in as root and start the crm tool:

   # crm configure

2. Configure the primitive. Change the intervals if needed:

   crm(live)configure# primitive myRSC ocf:myCorp:myAppl \
       op monitor interval=60 \
       op monitor interval=61 role=Master

3. Create the master slave resource:

   crm(live)configure# ms myRSC-clone myRSC

7.4. Managing Cluster Resources¶

Apart from the possibility to configure your cluster resources, the crm tool also allows you to manage existing resources. The following subsections gives you an overview.

7.4.1. Starting a New Cluster Resource¶

To start a new cluster resource you need the respective identifier. Proceed as follows:

1. Log in as root and start the crm tool:

   # crm configure

2. Switch to the resource level:

   crm(live)# resource

3. Start the resource with start and press the →| key to show all known resources:

   crm(live)resource# start start ID

7.4.2. Cleaning Up Resources¶

A resource will be automatically restarted if it fails, but each failure raises the resource's failcount. If a migration-threshold has been set for that resource, the node will no longer be allowed to run the resource as soon as the number of failures has reached the migration threshold.

1. Open a shell and log in as user root.

2. Get a list of all your resources:

   crm resource list
     ...
   Resource Group: dlm-clvm:1
            dlm:1  (ocf::pacemaker:controld) Started 
            clvm:1 (ocf::lvm2:clvmd) Started
            cmirrord:1     (ocf::lvm2:cmirrord) Started

3. Remove the resource:

   crm resource cleanup dlm-clvm

   For example, if you want to stop the DLM resource, from the dlm-clvm resource group, replace RSC with dlm.

7.4.3. Removing a Cluster Resource¶

Proceed as follows to remove a cluster resource:

1. Log in as root and start the crm tool:

   # crm configure

2. Run the following command to get a list of your resources:

   crm(live)# resource status

   For example, the output can look like this (whereas myIP is the relevant identifier of your resource):

   myIP    (ocf::IPaddr:heartbeat) ...

3. Delete the resource with the relevant identifier (which implies a commit too):

   crm(live)# configure delete YOUR_ID

4. Commit the changes:

   crm(live)# configure commit

7.4.4. Migrating a Cluster Resource¶

Although resources are configured to automatically fail over (or migrate) to other nodes of the cluster in the event of a hardware or software failure, you can also manually move a resource to another node in the cluster using either the Pacemaker GUI or the command line.

Use the migrate command for this task. For example, to migrate the resource ipaddress1 to a cluster node named node2, use these commands:

# crm resource
crm(live)resource# migrate ipaddress1 node2

7.5. Setting Passwords Independent of cib.xml¶

In case your cluster configuration contains sensitive information, such as passwords, it should be stored in local files. That way, these parameters will never be logged or leaked in support reports.

Before using secret, better run the show command first to get an overview of all your resources:

# crm configure show
primitive mydb ocf:heartbeat:mysql \
   params replication_user=admin ...

If you want to set a password for the above mydb resource, use the following commands:

# crm resource secret mydb set passwd linux
INFO: syncing /var/lib/heartbeat/lrm/secrets/mydb/passwd to [your node list]

You can get the saved password back with:

# crm resource secret mydb show passwd
linux

Note that the parameters need to be synchronized between nodes; the crm resource secret command will take care of that. We highly recommend to only use this command to manage secret parameters.

7.6. Retrieving History Information¶

Investigating the cluster history is a complex task. To simplify this task, the crm shell contains the history command with its subcommands. It is assumed SSH is configured correctly.

Each cluster moves states, migrates resources, or starts important processes. All these actions can be retrieved by subcommands of history. Alternatively, use Hawk as explained in Procedure 6.23, “Viewing Transitions with the History Explorer”.

By default, all history commands look at the events of the last hour. To change this time frame, use the limit subcommand. The syntax is:

# crm history
crm(live)history# limitFROM_TIME [TO_TIME]

Some valid examples include:

limit4:00pm , limit16:00

Both commands mean the same, today at 4pm.

limit2012/01/12 6pm

January 12th 2012 at 6pm

limit"Sun 5 20:46"

In the current year of the current month at Sunday the 5th at 8:46pm

Find more examples and how to create time frames at http://labix.org/python-dateutil.

The info subcommand shows all the parameters which are covered by the the hb_report:

crm(live)history# info
Source: live
Period: 2012-01-12 14:10:56 - end
Nodes: earth
Groups: 
Resources:

To limit hb_report to certain parameters view the available options with the subcommand help.

To narrow down the level of detail, use the subcommand detail with a level:

crm(live)history# detail 2

The higher the number, the more detailed your report will be. Default is 0 (zero).

After you have set above parameters, use log to show the log messages.

To display the last transition, use the following command:

crm(live)history# transition -1
INFO: fetching new logs, please wait ...

This command fetches the logs and runs dotty (from the graphviz package) to show the transition graph. The shell opens the log file which you can browse with the ↓ and ↑ cursor keys.

If you do not want to open the transition graph, use the nograph option:

crm(live)history# transition -1 nograph

7.7. For More Information¶

• The crm man page.

• See Highly Available NFS Storage with DRBD and Pacemaker (↑Highly Available NFS Storage with DRBD and Pacemaker) for an exhaustive example.