Deploy WebLogic12c to Multiple Machines

In this post, we set-up a WebLogic cluster that spans multiple machines. We start with installing the WebLogic software on the machines involved. Next, we create the WebLogic domain by using the WebLogic Scripting Tool (WLST). Use pack and unpack to create managed server directories on remote machines. Create start and stop scripts for the environment. Deploy an application and tune it by using deployment override descriptors. Set-up load balancing by using the Apache HTTP Server and show the steps involved in scaling the environment. Finally, we perform a load test by using The Grinder and use JRockit Mission Control to obtain insight in the performance of the cluster.

Install WebLogic

First, we choose an installation directory, for example /home/oracle/weblogic12.1.1. This will be our middleware home in the installation of WebLogic. To keep things simple use the same directory structure on both machines.

To install JRockit we follow these steps:

• Run the file jrrt-4.0.1-1.6.0-linux-x64.bin (JRockit can be downloaded here).
• Click next on the welcome screen.
• Define the install directory /home/oracle/jrrt-4.0.1-1.6.0 and click next.
• Optionally select extra components and click next.
• When the installation is finished click done.

To WebLogic we follow these steps:

• Navigate to the JDK’s bin directory (/home/oracle/jrrt-4.0.1-1.6.0/bin).
• Enter the following command: ./java -d64 -Xms1024m -Xmx1024m -jar wls1211_generic.jar (WebLogic can be downloaded here).
• Click next on the welcome screen.
• Define the Middleware-Home directory and click next.
• Fill in support credentials if applicable and click next.
• Select custom as installation type and click next.
• De-select the evaluation database and click next.
• Select the installed JDK (if the JDK is not shown, click browse and browse to the root of the JDK) and click next.
• Accept the defaults in the directories screen and click next.
• Click next to start the installation.
• De-select run quickstart and click done.

Create domain

To create our domain, we are going to use WLST. The following script, creates the domain in production mode, disables hostname verification, creates two managed server that are clustered and are present on different machines and creates the artifacts needed to run the application, such as JMS and a data source.

beahome = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
adminusername = 'weblogic';
adminpassword = 'magic12c';
adminservername='AdminServer';
adminserverurl='t3://172.31.0.113:7001';
domainname = 'base_domain';
domaindirectory = beahome + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domainname;
domaintemplate = beahome + pathseparator + 'wlserver_12.1' + pathseparator + 'common' + pathseparator + 'templates' + pathseparator + 'domains' + pathseparator + 'wls.jar';
jvmdirectory = '/home/oracle/jrrt-4.0.1-1.6.0';

print 'CREATE DOMAIN';
readTemplate(domaintemplate);
setOption('DomainName', domainname);
setOption('OverwriteDomain', 'true');
setOption('ServerStartMode', 'prod');
cd('/Security/base_domain/User/weblogic');
cmo.setName(adminusername);
cmo.setUserPassword(adminpassword);
cd('/');
writeDomain(domaindirectory);

# When using startServer command on WebLogic12c, we have to perform the following steps (1 and 2):
# Starting weblogic server ...
# WLST-WLS-1326447719560: Exception in thread "Main Thread" java.lang.AssertionError: JAX-WS 2.2 API is required, but an older version was found in the JDK.
# WLST-WLS-1326447719560: Use the endorsed standards override mechanism (http://java.sun.com/javase/6/docs/technotes/guides/standards/).
# WLST-WLS-1326447719560: 1) locate the bundled Java EE 6 endorsed directory in $WL_HOME/endorsed.
# WLST-WLS-1326447719560: 2) copy those JAR files to $JAVA_HOME/jre/lib/endorsed OR add the endorsed directory to the value specified by system property java.endorsed.dirs.
print 'START ADMIN SERVER';
startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword, adminserverurl);

print 'START EDIT MODE';
edit();
startEdit();

#print 'CHANGE NODEMANAGER USERNAME AND PASSWORD';
#cd('/SecurityConfiguration/' + domainname);
#cmo.setNodeManagerUsername(adminusername);
#set('NodeManagerPasswordEncrypted', encrypt(adminpassword, domaindirectory));
#cd('/');

print 'DISABLE HOSTNAME VERIFICATION';
cd('/Servers/' + adminservername + '/SSL/' + adminservername);
cmo.setHostnameVerificationIgnored(true);
cmo.setHostnameVerifier(None);
cmo.setTwoWaySSLEnabled(false);
cmo.setClientCertificateEnforced(false);
cd('/');

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'SHUTDOWN THE ADMIN SERVER';
shutdown(block='true');

print 'START ADMIN SERVER';
startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword, adminserverurl);

print 'START EDIT MODE';
edit();
startEdit();

print 'CREATE MACHINE: machine1';
machine1 = cmo.createUnixMachine('machine1');
machine1.setPostBindUIDEnabled(true);
machine1.setPostBindUID('oracle');
machine1.getNodeManager().setListenAddress('172.31.0.175');
machine1.getNodeManager().setNMType('ssl');

print 'CREATE MACHINE: machine2';
machine2 = cmo.createUnixMachine('machine2');
machine2.setPostBindUIDEnabled(true);
machine2.setPostBindUID('oracle');
machine2.getNodeManager().setListenAddress('172.31.0.113');
machine2.getNodeManager().setNMType('ssl');

print 'CREATE CLUSTER: CLUSTER';
cluster = cmo.createCluster('cluster');
cluster.setClusterMessagingMode('unicast');

print 'CREATE MANAGED SERVER: server1';
server1 = cmo.createServer('server1');
server1.setListenPort(7002);
server1.setListenAddress('172.31.0.175');
server1.setAutoRestart(true);
server1.setAutoKillIfFailed(true);
server1.setRestartMax(2);
server1.setRestartDelaySeconds(10);
server1.getServerStart().setJavaHome(jvmdirectory);
server1.getServerStart().setJavaVendor('Oracle');
server1.getServerStart().setArguments('-jrockit -Xms1024m -Xmx1024m -Xns256m -Xgc:throughput');

print 'CREATE MANAGED SERVER: server2';
server2 = cmo.createServer('server2');
server2.setListenPort(7003);
server2.setListenAddress('172.31.0.113');
server2.setAutoRestart(true);
server2.setAutoKillIfFailed(true);
server2.setRestartMax(2);
server2.setRestartDelaySeconds(10);
server2.getServerStart().setJavaHome(jvmdirectory);
server2.getServerStart().setJavaVendor('Oracle');
server2.getServerStart().setArguments('-jrockit -Xms1024m -Xmx1024m -Xns256m -Xgc:throughput');

print 'ADD MANAGED SERVERS TO CLUSTER';
server1.setCluster(cluster);
server2.setCluster(cluster);

print 'ADD MANAGED SERVERS TO MACHINE';
server1.setMachine(machine1);
server2.setMachine(machine2);

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'START EDIT MODE';
startEdit();

print 'CREATE FILESTORE FOR SERVER1';
filestore1 = cmo.createFileStore('FileStore1');
filestore1.setDirectory(beahome + pathseparator + 'deploy');
targets = filestore1.getTargets();
targets.append(server1);
filestore1.setTargets(targets);

print 'CREATE JMS SERVER FOR SERVER1';
jmsserver1 = cmo.createJMSServer('JMSServer1');
jmsserver1.setPersistentStore(filestore1);
jmsserver1.setTargets(targets);

targets.remove(server1);
targets.append(server2);

print 'CREATE FILESTORE FOR SERVER2';
filestore2 = cmo.createFileStore('FileStore2');
filestore2.setDirectory(beahome + pathseparator + 'deploy');
filestore2.setTargets(targets);

print 'CREATE JMS SERVER FOR SERVER2';
jmsserver2 = cmo.createJMSServer('JMSServer2');
jmsserver2.setPersistentStore(filestore2);
jmsserver2.setTargets(targets);

targets.remove(server2);
targets.append(cluster);

print 'CREATE JMS SYSTEM MODULE';
module = cmo.createJMSSystemResource('SystemModule');
module.setTargets(targets);

print 'CREATE SUBDEPLOYMENT';
module.createSubDeployment('SubDeployment');
cd('/JMSSystemResources/SystemModule/SubDeployments/SubDeployment');
set('Targets',jarray.array([ObjectName('com.bea:Name=JMSServer1,Type=JMSServer'), ObjectName('com.bea:Name=JMSServer2,Type=JMSServer')], ObjectName));
cd('/');

resource = module.getJMSResource();

print 'CREATE CONNECTION FACTORY';
resource.createConnectionFactory('ConnectionFactory');
connectionfactory = resource.lookupConnectionFactory('ConnectionFactory');
connectionfactory.setJNDIName('jms/ConnectionFactory');
connectionfactory.setDefaultTargetingEnabled(true);
connectionfactory.getTransactionParams().setTransactionTimeout(3600);
connectionfactory.getTransactionParams().setXAConnectionFactoryEnabled(true);

print 'CREATE UNIFORM DISTRIBUTED QUEUE';
resource.createUniformDistributedQueue('DistributedQueue');
distributedqueue = resource.lookupUniformDistributedQueue('DistributedQueue');
distributedqueue.setJNDIName('jms/CompanyQueue');
distributedqueue.setLoadBalancingPolicy('Round-Robin');
distributedqueue.setSubDeploymentName('SubDeployment');

print 'CREATE DATA SOURCE';
datasource = cmo.createJDBCSystemResource('DataSource');
datasource.setTargets(targets);
jdbcResource = datasource.getJDBCResource();
jdbcResource.setName('DataSource');
names = ['jdbc/exampleDS'];
dataSourceParams = jdbcResource.getJDBCDataSourceParams();
dataSourceParams.setJNDINames(names);
dataSourceParams.setGlobalTransactionsProtocol('LoggingLastResource');
driverParams = jdbcResource.getJDBCDriverParams();
driverParams.setUrl('jdbc:oracle:thin:@hostname:1521:sid');
driverParams.setDriverName('oracle.jdbc.OracleDriver');
driverParams.setPassword('password');
driverProperties = driverParams.getProperties();
driverProperties.createProperty('user');
userProperty = driverProperties.lookupProperty('user');
userProperty.setValue('username');
connectionPoolParams = jdbcResource.getJDBCConnectionPoolParams();
connectionPoolParams.setTestTableName('SQL SELECT 1 FROM DUAL');
connectionPoolParams.setConnectionCreationRetryFrequencySeconds(100);

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'SHUTDOWN THE ADMIN SERVER';
shutdown(block='true');

To run the script we can use the following (which also shows the output logging):

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/
[oracle@edu-wls-rh bin]$ ./wlst.sh /home/oracle/weblogic12.1.1/deploy/scripts/create_environment.py 

CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CREATE DOMAIN
START ADMIN SERVER
Starting weblogic server ...
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:51 AM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:51 AM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:52 AM CET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-x86_64 from Oracle Corporation.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:52 AM CET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.1.0  Wed Dec 7 08:40:57 PST 2011 1445491 >
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Notice> <Log Management> <BEA-170019> <The server log file /home/oracle/weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:57 AM CET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:00 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:00 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 172.31.0.113:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>
Server started successfully.
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START EDIT MODE
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit(). 

For more help, use help(edit)

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
DISABLE HOSTNAME VERIFICATION
SAVE AND ACTIVATE CHANGES
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141239> <The non-dynamic attribute HostnameVerificationIgnored on weblogic.management.configuration.SSLMBeanImpl@4c9421a2([base_domain]/Servers[AdminServer]/SSL[AdminServer]) has been changed. This may require redeploying or rebooting configured entities.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141238> <A non-dynamic change has been made which affects the server AdminServer. This server must be rebooted in order to consume this change.> 

The following non-dynamic attribute(s) have been changed on MBeans
that require server re-start:
MBean Changed : com.bea:Name=AdminServer,Type=SSL,Server=AdminServer
Attributes changed : HostnameVerificationIgnored

Activation completed
SHUTDOWN THE ADMIN SERVER
Shutting down the server AdminServer with force=false while connected to AdminServer ...
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SUSPENDING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <HTTP> <BEA-101278> <There are no active sessions. The Web service is ready to suspend.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SHUTTING_DOWN.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[3]", listening on 0:0:0:0:0:0:0:1:7001, was shut down.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default", listening on 172.31.0.113:7001, was shut down.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[2]", listening on 127.0.0.1:7001, was shut down.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[1]", listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001, was shut down.>
<Jan 16, 2012 9:12:11 AM CET> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:11 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>
WLST-WLS-1326701511109: Stopped draining WLST-WLS-1326701511109
WLST-WLS-1326701511109: Stopped draining WLST-WLS-1326701511109
WLST lost connection to the WebLogic Server that you were
connected to, this may happen if the server was shutdown or
partitioned. You will have to re-connect to the server once the
server is available.
Disconnected from weblogic server: AdminServer
Disconnected from weblogic server:
START ADMIN SERVER
Starting weblogic server ...
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-x86_64 from Oracle Corporation.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:16 AM CET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.1.0  Wed Dec 7 08:40:57 PST 2011 1445491 >
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Notice> <Log Management> <BEA-170019> <The server log file /home/oracle/weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:21 AM CET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:24 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:24 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 172.31.0.113:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>
Server started successfully.
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START EDIT MODE

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
CREATE MACHINE: machine1
CREATE MACHINE: machine2
CREATE CLUSTER: CLUSTER
CREATE MANAGED SERVER: server1
CREATE MANAGED SERVER: server2
ADD MANAGED SERVERS TO CLUSTER
ADD MANAGED SERVERS TO MACHINE
SAVE AND ACTIVATE CHANGES
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed
START EDIT MODE
Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
CREATE FILESTORE FOR SERVER1
CREATE JMS SERVER FOR SERVER1
CREATE FILESTORE FOR SERVER2
CREATE JMS SERVER FOR SERVER2
CREATE JMS SYSTEM MODULE
CREATE SUBDEPLOYMENT
CREATE CONNECTION FACTORY
CREATE UNIFORM DISTRIBUTED QUEUE
CREATE DATA SOURCE
SAVE AND ACTIVATE CHANGES
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed
SHUTDOWN THE ADMIN SERVER
Shutting down the server AdminServer with force=false while connected to AdminServer ...
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SUSPENDING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <HTTP> <BEA-101278> <There are no active sessions. The Web service is ready to suspend.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SHUTTING_DOWN.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default", listening on 172.31.0.113:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[2]", listening on 127.0.0.1:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[3]", listening on 0:0:0:0:0:0:0:1:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[1]", listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:34 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:32 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>
WLST-WLS-1326701534609: Stopped draining WLST-WLS-1326701534609
WLST-WLS-1326701534609: Stopped draining WLST-WLS-1326701534609
WLST lost connection to the WebLogic Server that you were
connected to, this may happen if the server was shutdown or
partitioned. You will have to re-connect to the server once the
server is available.
Disconnected from weblogic server: AdminServer
Disconnected from weblogic server:

Next, we add a boot.properties file in the ${DOMAIN_HOME}/servers/AdminServer/security directory:

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/
[oracle@edu-wls-rh AdminServer]$ mkdir security
[oracle@edu-wls-rh AdminServer]$ vi boot.properties

Add the following name-value pairs to the boot.properties file:

username=weblogic
password=magic12c

Note that when the admin server is started these values will be encrypted. The node manager requires authentication to start and stop managed servers. The first time the node manager is started it communicates with the admin server to obtain a username and password that will be used by the admin server to authenticate the node manager. As we created the domain in production mode, random node manager credits are created. When we want to use, for example, nmConnect we have to set the node manager username and password to known values. To this end start the admin server open the admin console:

• Click on base_domain, security, general and click on the advanced link.
• Edit the NodeManager Username and Password. In our case, we set these to respectively weblogic and magic12c.
• Start the node manager by using the startNodeManager.sh script.

WebLogic provides two command-line utilities, pack and unpack, that helps to create new managed server directories on remote machines, for example on the 172.31.0.113 machine we do the following (first navigate to the ${WL_HOME}/common/bin directory):

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/
[oracle@edu-wls-rh bin]$ ./pack.sh -managed=true -domain=/home/oracle/weblogic12.1.1/user_projects/domains/base_domain -template=/home/oracle/temp/base_domain.jar -template_name=base_domain
<< read domain from "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
>>  succeed: read domain from "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
<< set config option Managed to "true"
>>  succeed: set config option Managed to "true"
<< write template to "/home/oracle/temp/base_domain.jar"
....................................................................................................
>>  succeed: write template to "/home/oracle/temp/base_domain.jar"
<< close template
>>  succeed: close template

Next, we copy the created base_domain.jar to the 172.31.0.175 machine, for example, by using:

[oracle@edu-wls-rh bin]$ scp [email protected]:/home/oracle/temp/base_domain.jar [email protected]:/home/oracle/temp/
[email protected]'s password:
[email protected]'s password:
base_domain.jar                                                                                  100%   45KB  44.9KB/s   00:00
Connection to 172.31.0.113 closed.

To use unpack on the 172.31.0.175 machine we can do the following (first navigate to the ${WL_HOME}/common/bin directory):

[oracle@middleware-magic ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/
[oracle@middleware-magic bin]$ ./unpack.sh -domain=/home/oracle/weblogic12.1.1/user_projects/domains/base_domain -template=/home/oracle/temp/base_domain.jar
<< read template from "/home/oracle/temp/base_domain.jar"
>>  succeed: read template from "/home/oracle/temp/base_domain.jar"
<< set config option DomainName to "base_domain"
>>  succeed: set config option DomainName to "base_domain"
<< write Domain to "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
...............................................................................................
>>  succeed: write Domain to "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
<< close template
>>  succeed: close template

To test the set-up, start the admin server (by using ./startWebLogic.sh located in the ${DOMAIN_HOME} directory). Subsequently, start the node managers on both machines (by using ./startNodeManager.sh located in the ${WL_HOME}/server/bin directory). When the node manager is started for the first time a nodemanager.properties file is created in the node manager home directory (which is ${WL_HOME}/common/nodemanager by default). We edit the nodemanager.properties and set the property StartScriptEnabled=false (the default is true). When doing this, we need to copy the ${WL_HOME}/endorsed directory to the jrrt-4.0.1-1.6.0/jre/lib/endorsed directory. When the changes are made the node manager needs to be restarted. Subsequently, use the admin console to the start the managed servers.

Start and stop scripts

In general, it is recommended to start the node manager when the machine boots. In this case, we need to know where to put our custom commands that will be called when the system boots. Note that RedHat Enterprise Linux has a /etc/rc.d/rc.local file, that can be used to put custom commands in. Let us do it in the recommended script-based way. Note that Unix-based systems specify so-called run levels, and that for each run level, scripts can be defined that start a certain service. These scripts are located in the /etc/rc.d/init.d directory. This allows for services to be started when the system boots or to be stopped on system shutdown. The different run levels are specified by a specific directory in the /etc/rc.d directory, i.e.,

• rc0.d – contains scripts that are executed on system shutdown.
• rc1.d – contains scripts for single-user mode.
• rc2.d – contains scripts for multi-user mode.
• rc3.d – contains scripts for multi-user mode and networking.
• rc4.d – not used.
• rc5.d – same as rc3.d plus some graphical stuff.
• rc6.d – contains scripts that are executed on system reboot.

The boot sequence is as follows: in the /etc/inittab file the starting runlevel is defined, the script /etc/rc.d/rc.sysinit is called and /etc/rc.d/rc is run. The rc script looks in the /etc/rc.d/rc<start-runlevel>.d to execute the K**<script-name> scripts with the stop option. After this the S**<script-name> scripts are executed with the start option. Note that scripts are started in numerical order, i.e., the S10network script is executed before the S80sendmail script.

To create a node manager ‘service’, we first create basic start and stop scripts for the node manager. To this end, we will use WLST. To start the node manager by using WLST we can use the following:

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
listen_address = '172.31.0.175';
nodemanager_listen_port = '5556';
domain_name='base_domain';

node_manager_home = bea_home + pathseparator + 'wlserver_12.1' + pathseparator + 'common' + pathseparator + 'nodemanager';

startNodeManager(verbose='true', NodeManagerHome=node_manager_home, ListenPort=nodemanager_listen_port, ListenAddress=listen_address);

To stop the node manager by using WLST we can use the following:

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
admin_username = 'weblogic';
admin_password = 'magic12c';
listen_address = '172.31.0.175';
nodemanager_listen_port = '5556';
admin_server_listen_port = '7001';
domain_name = 'base_domain';

domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;
admin_server_url='t3://' + listen_address + ':' + admin_server_listen_port;

nmConnect(admin_username, admin_password, listen_address, nodemanager_listen_port, domain_name, domain_home, 'ssl');
stopNodeManager();

Create two shell scripts that respectively run the start and stop script, i.e.,

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/startNodeManager.py

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/stopNodeManager.py

Create a boot script (nodemanager) under the root user and place this script in the /etc/rc.d/init.d directory. An example boot script for the node manager looks as follows:

#!/bin/sh
#
# chkconfig: 235 91 35
# description: starts and stops the node manager
#
#
. /etc/rc.d/init.d/functions

RETVAL=0
SERVICE="nodemanager"

start() {
	echo "Starting Node Manager"
	su - oracle -c "/home/oracle/weblogic12.1.1/deploy/scripts/NodeManagerStartService.sh" >/dev/null 2>&1
	RETVAL=$?
	[ $RETVAL -eq 0 ] && success || failure
	echo
	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/${SERVICE}
	return $RETVAL
} 

stop() {
	echo "Stopping Node Manager"
	su - oracle -c "/home/oracle/weblogic12.1.1/deploy/scripts/NodeManagerStopService.sh" >/dev/null 2>&1
	RETVAL=$?
	[ $RETVAL -eq 0 ] && success || failure
	echo
	[ $RETVAL -eq 0 ] && rm -r /var/lock/subsys/${SERVICE}
	return $RETVAL
} 

restart() {
	stop
	start
} 

case "$1" in
	start)
		start
		;;
	stop)
		stop
		;;
	restart)
		restart
		;;
	*)
		echo $"Usage: $0 {start|stop|restart}"
		exit 1
esac 

exit $?

By using the chkconfig command we can update the runlevel information for system services, for example, chkconfig --add nodemanager. To test the set-up shut the system down and start it again. To check if the nodemanager is running we can use either ps -ef|grep java or netstat -anp|grep :5556 which assumes the node manager is listening on port 5556.

By using WLST we can also create start and stop scripts for the admin server and the managed servers, for example,

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
admin_username = 'weblogic';
admin_password = 'magic12c';
listen_address_machine1 = '172.31.0.175';
listen_address_machine2 = '172.31.0.113';
nodemanager_listen_port = '5556';
admin_server_listen_port = '7001';
domain_name = 'base_domain';

domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;
admin_server_url='t3://' + listen_address_machine2 + ':' + admin_server_listen_port;

print 'CONNECT TO NODE MANAGER ON MACHINE2';
nmConnect(admin_username, admin_password, listen_address_machine2, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'START ADMIN SERVER ON MACHINE2';
nmStart('AdminServer');

print 'CONNECT TO ADMIN SERVER';
connect(admin_username, admin_password, admin_server_url);

print 'START MANAGED SERVERS ON MACHINE2';
start('server2','Server');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';
nmDisconnect();

print 'CONNECT TO NODE MANAGER ON MACHINE1';
nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'START MANAGED SERVERS ON MACHINE1';
start('server1','Server');
# server3 will be added in a later stadium when we scale the cluster
#start('server3','Server');

print 'DISCONNECT FROM THE ADMIN SERVER';
disconnect();

print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';
nmDisconnect();

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
admin_username = 'weblogic';
admin_password = 'magic12c';
listen_address_machine1 = '172.31.0.175';
listen_address_machine2 = '172.31.0.113';
nodemanager_listen_port = '5556';
admin_server_listen_port = '7001';
domain_name = 'base_domain';

domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;
admin_server_url='t3://' + listen_address_machine2 + ':' + admin_server_listen_port;

print 'CONNECT TO NODE MANAGER ON MACHINE1';
nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'CONNECT TO ADMIN SERVER';
connect(admin_username, admin_password, admin_server_url);

print 'STOPPING MANAGED SERVERS ON MACHINE1';
shutdown('server1','Server','true',1000,'true');
# server3 will be added in a later stadium when we scale the cluster
#shutdown('server3','Server','true',1000,'true');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';
nmDisconnect();

print 'CONNECT TO NODE MANAGER ON MACHINE2';
nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'STOPPING MANAGED SERVERS ON MACHINE2';
shutdown('server2','Server','true',1000,'true');

print 'STOPPING ADMIN SERVER ON MACHINE2';
shutdown('AdminServer','Server','true',1000,'true');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';
nmDisconnect();

For these WLST scripts, we create two shell scripts that respectively run the start and stop script, i.e.,

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/startDomain.py

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/stopDomain.py

To start the domain we can use:

[oracle@edu-wls-rh scripts]$ ./DomainStartService.sh 

CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CONNECT TO NODE MANAGER ON MACHINE2
Connecting to Node Manager ...
<Jan 16, 2012 1:59:54 PM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Jan 16, 2012 1:59:54 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
Successfully Connected to Node Manager.
START ADMIN SERVER ON MACHINE2
Starting server AdminServer ...
Successfully started server AdminServer ...
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START MANAGED SERVERS ON MACHINE2

Starting server server2 ...............................................................
Server with name server2 started successfully
DISCONNECT FROM NODE MANAGER ON MACHINE2
Successfully disconnected from Node Manager.
CONNECT TO NODE MANAGER ON MACHINE1
Connecting to Node Manager ...
Successfully Connected to Node Manager.
START MANAGED SERVERS ON MACHINE1

Starting server server1 ..................................................................................
Server with name server1 started successfully
DISCONNECT FROM THE ADMIN SERVER
Disconnected from weblogic server: AdminServer
DISCONNECT FROM NODE MANAGER ON MACHINE1
Successfully disconnected from Node Manager.

To stop the domain we can use:

[oracle@edu-wls-rh scripts]$ ./DomainStopService.sh 

CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CONNECT TO NODE MANAGER ON MACHINE1
Connecting to Node Manager ...
<Jan 16, 2012 2:05:40 PM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Jan 16, 2012 2:05:40 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
Successfully Connected to Node Manager.
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

STOPPING MANAGED SERVERS ON MACHINE1
Shutting down the server server1 with force=true while connected to AdminServer ...
DISCONNECT FROM NODE MANAGER ON MACHINE1
Successfully disconnected from Node Manager.
CONNECT TO NODE MANAGER ON MACHINE2
Connecting to Node Manager ...
Successfully Connected to Node Manager.
STOPPING MANAGED SERVERS ON MACHINE2
Shutting down the server server2 with force=true while connected to AdminServer ...
STOPPING ADMIN SERVER ON MACHINE2
Shutting down the server AdminServer with force=true while connected to AdminServer ...
WLST lost connection to the WebLogic Server that you were
connected to, this may happen if the server was shutdown or
partitioned. You will have to re-connect to the server once the
server is available.
Disconnected from weblogic server: AdminServer
Disconnected from weblogic server:
DISCONNECT FROM NODE MANAGER ON MACHINE2
Successfully disconnected from Node Manager.

Deployment

The application can be found in the post WebLogic12c in Action (in the ‘My first JavaEE6 application’ section). First, we create the following directory structure in /home/oracle/weblogic12.1.1/deploy:

/loadtest
	/app
		LoadTest6.ear
	/plan

By using weblogic.PlanGenerator, we generate a deployment plan:

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/server/bin/
[oracle@edu-wls-rh bin]$ . ./setWLSEnv.sh
CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/home/oracle/weblogic12.1.1/wlserver_12.1/server/bin:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/bin:/home/oracle/jrrt-4.0.1-1.6.0/jre/bin:/home/oracle/jrrt-4.0.1-1.6.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/bin

Your environment has been set.
[oracle@edu-wls-rh bin]$ java weblogic.PlanGenerator -plan /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Plan.xml -root /home/oracle/weblogic12.1.1/deploy/loadtest/ /home/oracle/weblogic12.1.1/deploy/loadtest/app/LoadTest6.ear
Generating plan for application /home/oracle/weblogic12.1.1/deploy/loadtest/app/LoadTest6.ear
Export option is: dependencies
Exporting properties...
Saving plan to /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Plan.xml...
<Jan 17, 2012 1:56:43 PM CET> <Info> <J2EE Deployment SPI> <BEA-260086> <The descriptor information at /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Web.war/WEB-INF/weblogic.xml was saved to the configuration area.>
<Jan 17, 2012 1:56:43 PM CET> <Info> <J2EE Deployment SPI> <BEA-260072> <Saved configuration for application, LoadTest6.ear>

When the deployment plan has been generated, the files Plan.xml and weblogic.xml are created automatically. The other files, weblogic-application.xml and weblogic-ejb-jar.xml, we have to create ourselves according to the below directory structure:

/loadtest
	/app
		LoadTest6.ear
	/plan
		/LoadTest6.ear
			/META-INF
				weblogic-application.xml
		/Model.jar
			/META-INF
				weblogic-ejb-jar.xml
		/Web.war
			/WEB-INF
				weblogic.xml
		Plan.xml

in which the deployment overrides (weblogic-application.xml, weblogic-ejb-jar.xml and weblogic.xml) have the following contents:

<weblogic-application xmlns="http://xmlns.oracle.com/weblogic/weblogic-application" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-application http://xmlns.oracle.com/weblogic/weblogic-application/1.4/weblogic-application.xsd">
</weblogic-application>

<weblogic-ejb-jar xmlns="http://xmlns.oracle.com/weblogic/weblogic-ejb-jar" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-ejb-jar http://xmlns.oracle.com/weblogic/weblogic-ejb-jar/1.1/weblogic-ejb-jar.xsd">
	<weblogic-enterprise-bean>
		<ejb-name>Company</ejb-name>
		<enable-call-by-reference>True</enable-call-by-reference>
	</weblogic-enterprise-bean>
</weblogic-ejb-jar>

<weblogic-web-app xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.4/weblogic-web-app.xsd">
</weblogic-web-app>

The Java EE specification requires that EJB components invoked through their remote interfaces must use pass-by-value semantics, meaning that method parameters are copied during the invocation. Changes made to a parameter in the bean method are not reflected in the caller’s version of the object. Copying method parameters is required in the case of a true remote invocation, of course, because the parameters are serialized by the underlying RMI infrastructure before being provided to the bean method. Pass-by-value semantics are also required between components located in different enterprise applications in the same Java virtual machine due to classloader constraints. EJB components located in the same enterprise application archive (.ear) file are loaded by the same classloader and have the option of using pass-by-reference semantics for all invocations, eliminating the unnecessary copying of parameters passed during the invocation and improving performance. By setting the enable-call-by-reference parameter to true in weblogic-ejb-jar.xml, we enable this feature a specific bean in the application. Local references always use pass-by-reference semantics and are unaffected by the enable-call-by-reference setting. When we deploy an EJB with a remote interface and do not enable call by reference, WebLogic will issue a warning of the performance cost, i.e.,

<Jan 16, 2012 5:01:56 PM CET> <Warning> <EJB> <BEA-010202> <Call-by-reference is not enabled for EJB Company. The server will have better performance if it is enabled. To enable call-by-reference, set the enable-call-by-reference element to True in the weblogic-ejb-jar.xml deployment descriptor or corresponding annotation for this EJB.>

Load balancing

To install the Apache HTTP Server we can follow these steps:

• Unpack the httpd-2.2.21.tar.gz file:
  • gzip -d httpd-2.2.21.tar.gz
  • tar xvf httpd-2.2.21.tar
  • cd httpd-2.2.21
• The next step is to configure:
  • ./configure --prefix=/home/oracle/weblogic12.1.1/apache
• Next, compile the various parts for the Apache HTTP Server by using:
  • make
• To install the Apache HTTP Server we use:
  • make install
• Open the httpd.conf (/home/oracle/weblogic12.1.1/apache/conf) file and adjust the following directives:
  • Listen 7777
  • ServerName 172.31.0.113
• To test the set-up, start the Apache HTTP Server:
  • Navigate to /home/oracle/weblogic12.1.1/apache/bin.
  • Run: ./apachectl -k start (To stop the Apache HTTP Server we can use: ./apachectl -k stop).
  • Open a browser en type the following URL: http://172.31.0.113:7777.

To install the the WebLogic plugin we can follow these steps:

• The WebLogic plugins can be downloaded here.
• Unzip the distribution that is appropriate for the operating system, for example, WLSPlugin11g-64bitApache2.2-linux64-x86_64.zip.
• Copy the libclntsh.so.11.1, libnnz11.so and libwlssl.so files to the ${APACHE_HOME}/lib directory. Note in our case ${APACHE_HOME} is /home/oracle/weblogic12.1.1/apache.
• Copy the mod_wl.so file to the ${APACHE_HOME}/modules directory.
• Create a configuration file (for example, mod_wl.conf) for the WebLogic plugin and place this file in the ${APACHE_HOME}/conf directory.

The contents of the mod_wl.conf look as follows:

LoadModule weblogic_module   "/home/oracle/weblogic12.1.1/apache/modules/mod_wl.so"

<IfModule weblogic_module>
	ConnectTimeoutSecs 10
	ConnectRetrySecs 2
	DebugConfigInfo ON
	WLSocketTimeoutSecs 2
	WLIOTimeoutSecs 300
	Idempotent ON
	FileCaching ON
	KeepAliveSecs 20
	KeepAliveEnabled ON
	DynamicServerList ON
	WLProxySSL OFF
</IfModule>

<Location /LoadTest6>
	SetHandler weblogic-handler
	WebLogicCluster 172.31.0.175:7002,172.31.0.113:7003
</Location>

To let the Apache HTTP Server pick up the configuration we add the following to the httpd.conf file:

# put it near the end of the file where all the other includes are present
# mod_wl configuration
Include conf/mod_wl.conf

Restart the HTTP Server (./apachectl -k stop and ./apachectl -k start).

We are not happy with the load (do not know why, just a hypotheses) and need an extra server. We have two scaling options: vertical and horizontal scaling. Vertical scaling relates to adding more CPUs to a machine. To better utilize the server hardware we can add more WebLogic instances to the machine that could lead to increased application throughput. To determine if this is indeed the case we need to benchmark. Benchmarking for scalability is about measuring resource utilization. Good scalability means that service levels can be maintained while the workload is increased. If an application does not scale well, it is not fully utilizing the hardware. Consequently, throughput will degrade. Ideally, a linear load increase should lead to a linear degradation in service levels and performance. Linear scalability can be approached when so-called share nothing clusters are used. The nodes provide the same functionality and know nothing about other nodes in the cluster (no HTTP session replication). In this case, the computing ability of the cluster increases almost linearly as more nodes are added to the cluster, if the back-end information systems, such as a database, are powerful enough.

Applications that ‘share nothing’ are usually sharing state through the database. The application-tier can scale as far as when the database becomes a bottleneck. In general, relying on a single shared resource will eventually cause contention for that resource and thus limit the scalability. Caching is a good resolution. When we cache data at the application-tier we avoid calls to the database (and also avoid relational data to object data conversions). Caching solutions, such as Coherence, provide different kind of caching, i.e., replicated and partitioned. Replicated does not scale well when cache writes are involved as the data needs to be replicated across all the nodes in the grid. A partioned cache, on the other hand, scales very well when cache writes are involved as data ownership is spread throughout the cluster (the system automatically rebalances the data when the number of nodes in the grid changes – we do not need to decide on how to partition the data, it comes out of the box). Another plus is that access to the cache means at most one network trip, this in order to maintain linear scalability. An optimization on read-access can be made when data can be obtained locally (sticky access) in this case a hybrid solution such as the near cache can be applied.

Horizontal scaling relates to adding more machines to the environment, which gives a failover capability that we cannot get with vertical scaling. A good approach is to combine both scaling techniques to obtain better CPU utilization and failover capability.

In order to create a new server based on server1 open the admin console:

• Click environment, servers, select server1 and click clone.
• Change the server name to server3 and listen port to 7004.
• Click OK.

Create a JMS server and a file store and target these to server3:

• Click services, messaging, JMS servers.
• Click new and enter the name as JMSServer3.
• Click create a new store:
  • Choose file store as type and click next.
  • Enter the following parameters:
    • name: FileStore3
    • target: server3
    • directory: /home/oracle/weblogic12.1.1/deploy
  • Click OK.
• Click next and choose as target, server3.
• Click finish.

Add the created JMS server to the subdeployment of the JMS module:

• Click services, messaging, JMS modules and choose systemmodule.
• Click the subdeployments tab and choose subdeployment.
• Add JMSServer3 and click save.

Start server3 by using the admin console:

• Click environment, servers.
• Click the control tab.
• Select server3 and click start.

To test the configuration we can use the URL: http://172.31.0.113:7777/LoadTest6/?__WebLogicBridgeConfig. When the URL is accessed the General Server List should show the servers in the cluster, i.e.,

WebLogic Server Plugin version 1.1, <WLSPLUGINS_11.1.1.4.0_LINUX.X64_101209.1115>
Query String: '?__WebLogicBridgeConfig'
WebLogic Cluster List:
General Server List:
Host: '172.31.0.175' Port: 7004 SecurePort: 0 Status: OK
Host: '172.31.0.113' Port: 7003 SecurePort: 0 Status: OK
Host: '172.31.0.175' Port: 7002 SecurePort: 0 Status: OK
ConnectRetrySecs: '2'
ConnectTimeoutSecs: '10'
WLCookieName: JSESSIONID
Debug: 'no debugging'
DebugConfigInfo: 'ON'
DefaultFileName: ''
DisableCookie2Server: OFF
DynamicServerList: 'ON'
ErrorPage: ''
FileCaching: ON
Idempotent: ON
KeepAliveEnabled: ON
KeepAliveSecs: 20
MaxPostSize: '0'
MaxSkipTime: '10'
PathPrepend: ''
PathTrim: ''
QueryFromRequest: OFF
WLForwardUriUnparsed: OFF
WLAllowDoubleEscapedURI: OFF
SecureProxy: 'OFF'
StatPath: 'false'
WLDNSRefreshInterval: '0'
WLIOTimeoutSecs(old name is HungServerRecoverSecs): '300'
WLLogFile: ''
WLSocketTimeoutSecs: '2'
WLProxySSL: OFF
WLProxyPassThrough: OFF
WLProxySSLPassThrough: OFF

Runtime statistics:
requests: 0
successful requests: 0
Exception objects created: 0
Exception Objects deleted: 0
URL Objects created: 1
URL Objects deleted: 0
connections recycled: 0
UNKNOWN_ERROR_CODE exceptions: 0
CONNECTION_REFUSED exceptions: 0
CONNECTION_TIMEOUT exceptions: 0
READ_ERROR_FROM_CLIENT exceptions: 0
READ_ERROR_FROM_SERVER exceptions: 0
READ_ERROR_FROM_FILE exceptions: 0
WRITE_ERROR_TO_CLIENT exceptions: 0
WRITE_ERROR_TO_SERVER exceptions: 0
WRITE_ERROR_TO_FILE exceptions: 0
READ_TIMEOUT exceptions: 0
WRITE_TIMEOUT exceptions: 0
UNKNOWN_HOST exceptions: 0
NO_RESOURCES exceptions: 0
PROTOCOL_ERROR exceptions: 0
CONFIG_ERROR exceptions: 0
FAILOVER_REQUIRED exceptions: 0
POST_TIMEOUT exceptions: 0
REQUEST_ENTITY_TOO_LARGE exceptions: 0
HALF_OPEN_SOCKET_RETRY exceptions: 0
BAD_REQUEST_FROM_CLIENT exceptions: 0
SSL_INIT_ERROR exceptions: 0
Build date/time: Dec 20 2010 22:11:08
Change Number: 1013

The runtime statistics depict results when a number of requests (http://172.31.0.113:7777/LoadTest6/testservlet) are made. The new server is added to the general server list (which is due to the used DynamicServerList configuration). As the application is stateless and no sessions are created (no users need to login), we have no session binding. To get an idea how the load is balanced, click in the admin console on deployments, LoadTest6 and subsequently click on the workload, monitoring tab.

Testing

We can test the environment by using the The Grinder. To set-up The Grinder, we create the following scripts:

A script to set the environment (setGrinderEnv.sh):

#!/bin/sh
GRINDERPATH=/home/oracle/grinder-3.4
export GRINDERPATH

JAVA_HOME=/home/oracle/jdk1.6.0_24
export JAVA_HOME

USER_MEM_ARGS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC"
export USER_MEM_ARGS
GRINDERPROPERTIES=${GRINDERPATH}/examples/grinder.properties
export GRINDERPROPERTIES
CLASSPATH=${GRINDERPATH}/lib/grinder.jar:${CLASSPATH}
export CLASSPATH

PATH=${JAVA_HOME}/bin:${PATH}
export PATH

A script to start the console (startConsole.sh):

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Console

A script to start an agent that starts the test processes (startAgent.sh):

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Grinder ${GRINDERPROPERTIES}

The grinder.properties file has, among others, the following properties:

# The file name of the script to run.
#
# Relative paths are evaluated from the directory containing the
# properties file. The default is "grinder.py".
grinder.script = test.py

# The number of worker processes each agent should start. The default
# is 1.
grinder.processes = 1

# The number of worker threads each worker process should start. The
# default is 1.
grinder.threads = 1

# The number of runs each worker process will perform. When using the
# console this is usually set to 0, meaning "run until the console
# sneds a stop or reset signal". The default is 1.
grinder.runs = 0

The test.py script has the following contents:

from net.grinder.script.Grinder import grinder
from net.grinder.script import Test
from net.grinder.plugin.http import HTTPRequest

test1 = Test(1, "Request resource")
request1 = test1.wrap(HTTPRequest())

class TestRunner:
    def __call__(self):
        result = request1.GET("http://172.31.0.113:7777/LoadTest6/testservlet")

Start the test by running startConsole.sh and subsequently startAgent.sh:

During the loadtest we are interested in monitoring the paging. The Linux memory handler manages the allocation of physical memory by freeing portions of physical memory when possible. All processes use memory, but each process does not need all its allocated memory all the time. Taking advantage of this fact, the kernel frees up physical memory by writing some or all of a process’ memory to disk until it is needed again. The kernel uses paging and swapping to perform this memory management. Paging refers to writing portions (pages) of a process’ memory to disk. Swapping refers to writing the entire process to disk. When pages are written to disk, the event is called a page-out, and when pages are returned to physical memory, the event is called a page-in. A page fault occurs when the kernel needs a page, finds it does not exist in physical memory because it has been paged-out, and re-reads it in from disk. When the kernel detects that memory is running low, it attempts to free up memory by paging out. Though this may happen briefly from time to time, if page-outs are plentiful and constant, the kernel can reach a point where it is actually spending more time managing paging activity than running the applications, and system performance suffers. To monitor paging we can use, for example, vmstat 60 10 (which runs vmstat with ten updates, 60 seconds apart):

# Output machine1 (where server1 and server3 are running)
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 1185576  40744 521408    0    0    63    10  384  347  6  3 90  1  0
 2  0      0 1178368  40852 527824    0    0     0   132 2921 2509 21 10 68  0  0
 1  0      0 1174384  40924 530588    0    0     0    63 2442 2130 17  9 74  0  0
 0  0      0 1169796  41020 534336    0    0     0    77 2798 2408 19 10 71  0  0
 0  0      0 1162728  41124 540188    0    0     1   130 2941 2614 18 11 70  0  0
 1  0      0 1155028  41216 544020    0    0     0    76 2899 2549 21 10 69  0  0
 0  0      0 1150812  41316 548804    0    0     0   105 3130 2790 21 11 68  0  0
 0  0      0 1140256  41420 554224    0    0     0   120 2947 2595 19 11 69  0  0
 1  0      0 1121904  41528 561132    0    0     0   140 2896 2557 19 11 70  0  0
 3  0      0 1110760  41628 567212    0    0     0   123 2963 2657 18 13 68  0  0

Output machine2 (where the admin server and server2 are running)
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1622200  34556 574508    0    0    86    15  642  305  2  0 97  1  0
 0  0      0 1617552  34656 578236    0    0     0    87 2685 1967  5  2 93  0  0
 0  0      0 1615476  34748 580480    0    0     0    55 2309 1562  3  1 95  0  0
 0  0      0 1612872  34844 582992    0    0     0    64 2616 1898  4  2 95  0  0
 1  0      0 1609964  34948 585576    0    0     0    73 2735 2017  4  2 94  0  0
 1  0      0 1605508  35052 589116    0    0     0    81 2712 1985  4  2 94  0  0
 1  0      0 1602076  35152 592352    0    0     0    77 2913 2313  5  2 93  0  0
 0  0      0 1596448  35260 595104    0    0     0    69 2773 2095  4  2 94  0  0
 0  0      0 1590000  35360 599212    0    0     0    92 2693 2038  4  2 94  0  0
 0  0      0 1587164  35456 601748    0    0     0   106 2633 1861  5  2 93  0  0

 FIELD DESCRIPTION FOR VM MODE
   Procs
       r: The number of processes waiting for run time.
       b: The number of processes in uninterruptible sleep.

   Memory
       swpd: the amount of virtual memory used.
       free: the amount of idle memory.
       buff: the amount of memory used as buffers.
       cache: the amount of memory used as cache.
       inact: the amount of inactive memory. (-a option)
       active: the amount of active memory. (-a option)

   Swap
       si: Amount of memory swapped in from disk (/s).
       so: Amount of memory swapped to disk (/s).

   IO
       bi: Blocks received from a block device (blocks/s).
       bo: Blocks sent to a block device (blocks/s).

   System
       in: The number of interrupts per second, including the clock.
       cs: The number of context switches per second.

   CPU
       These are percentages of total CPU time.
       us: Time spent running non-kernel code. (user time, including nice time)
       sy: Time spent running kernel code. (system time)
	   id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
       wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
       st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

The values for si and so are both zero, indicating there are no page-ins and page-outs.

Let us look at some of the diagnostics collected by WebLogic. To this end, open the admin console, click deployments and click the monitoring tab. This page displays monitoring information for all applications deployed to the domain. The JMS tab displays monitoring information for all JMS destinations (note that the application only sends a message when a new person is added):

JMSServer1!JMSServer1.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40
JMSServer2!JMSServer2.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40
JMSServer3!JMSServer3.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40
SystemModule!JMSServer1@DistributedQueue - Bytes Received Count: 719274, Messages Received Count: 2347
SystemModule!JMSServer2@DistributedQueue - Bytes Received Count: 703990, Messages Received Count: 2297
SystemModule!JMSServer3@DistributedQueue - Bytes Received Count: 715648, Messages Received Count: 2335

The EJB (stateless and message-driven) tab displays monitoring information for all the Enterprise JavaBeans (EJBs):

Company	example - Pooled Beans Current Count: 3, Access Total Count: 348936, Transactions Committed Total Count: 348899, Transactions Rolled Back Total Count: 37
CompanyMDB example - Access Total Count: 6979, Processed Message Count: 2347, Transactions Committed Total Count: 6979

The JDBC tab displays monitoring information for all JDBC data sources:

DataSource server1 - Active Connections High Count: 2, Connection Delay Time: 155, PrepStmt Cache Access Count: 105396, Reserve Request Count (cummulative running count of requests for a connection): 116715, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0
DataSource server2 - Active Connections High Count: 2, Connection Delay Time: 127, PrepStmt Cache Access Count: 10173, Reserve Request Count (cummulative running count of requests for a connection): 116716, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0
DataSource server3 - Active Connections High Count: 2, Connection Delay Time: 143, PrepStmt Cache Access Count: 99600, Reserve Request Count (cummulative running count of requests for a connection): 116698, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0

The workload tab shows statistics for the Work Managers, constraints, and policies that are configured for application deployments:

default server1 - Pending Requests: 0, Completed Requests: 118664
default server2 - Pending Requests: 0, Completed Requests: 118616
default server3 - Pending Requests: 0, Completed Requests: 118635

We use the JRockit Mission Control Flight Recording, to see if there are any hick-ups due to garbage collections:

One thing to note is that the garbage collections did not run at same time:

• server1 (172.31.0.175): 03:46:18 154ms – 03:46:18 295ms
• server2 (172.31.0.113): 03:43:29 508ms – 03:43:29 751ms
• server3 (172.31.0.175): 03:43:52 399ms – 03:43:52 650ms

In general, JVM instances running on the same machine will typically not run the garbage collection at the same time. This means that we will have a JVM available to process application requests on other available CPUs. This is an advantage of vertical scaling that leads to a higher application throughput.

To get some insight what effect the garbage collection had on the application we use the events environment of the flight recording. We enable the following events:

• Java application – Java blocked
• Java application – Java wait
• Java virtual machine – GC – garbage collection
• WebLogic Server – EJB – EJB business method invocation
• WebLogic Server – JDBC – JDBC statement execute
• WebLogic Server – Servlet – Servlet invocation

The thread group ‘thread group for queue: weblogic.socket.Muxer’ contain the muxer threads. As a native muxer is used and we have 2 CPUs available, the number of threads is equal to #CPUs + 1 = 3. These threads are always showing this behavior, i.e., one thread at a time is active; picking requests of the sockets and put it in the execute queue. Here it will be picked up by an execute thread from the thread group ‘pooled threads’, which will process it, i.e., the servlet invocation (light green), the EJB business method invocation (blue) and the JDBC statement execution (purple). Note that during the garbage collection the execute threads have to wait (yellow) and a pause in the processing of the work defined for the execute thread is introduced.

References

[1] WebLogic 12c Documentation Library.
[2] Monitoring Virtual Memory with vmstat.

load test, multiple machines, scaling, WebLogic cluster