Running CCSM4

Reprinted from http://wiki.scinethpc.ca/wiki/index.php/Running_CCSM4

 

It is important to point out that all updates to the model system will only occur with CESM1.0 updates, not with CCSM4.0. It is also important to note that CCSM4 is a subset of CESM1. Although CESM1 supersedes CCSM4, users can run all CCSM4 experiments from the CESM1 code base.

 

Note: To run interactively on GPC interactive queue or TCS on small subset of interactive processors, see the instructions below.

The scientifically validated CESM1 runs are found in the list below (including a complete list of the model resolutions):

 

/project/ccsm/cesm1_current/scripts/create_newcase --list

 

  CESM1.0 README
 
For both a quick start as well as a detailed summary of creating and running 
a CESM model case, see the CESM1.0 User's Guide at
http://www.cesm.ucar.edu/models/cesm1.0
 
IMPORTANT INFORMATION ABOUT SCIENTIFIC VALIDATION
 
  CESM1.0 has the flexibility to configure cases with many different 
  combinations of component models, grids, and model settings, but this 
  version of CESM has only been validated scientifically for the following 
  fully active configurations:

     1.9x2.5_gx1v6   B_1850_CN
     1.9x2.5_gx1v6   B_1850_RAMPCO2_CN
     1.9x2.5_gx1v6   B_1850-2000_CN

     1.9x2.5_gx1v6   B_1850_CAM5

     0.9x1.25_gx1v6  B_1850_CN
     0.9x1.25_gx1v6  B_1850_RAMPCO2_CN
     0.9x1.25_gx1v6  B_1850-2000_CN

     0.9x1.25_gx1v6  B_1850_BGC-BPRP
     0.9x1.25_gx1v6  B_1850_BGC-BDRD
     0.9x1.25_gx1v6  B_1850-2000_BGC-BPRP
     0.9x1.25_gx1v6  B_1850-2000_BGC-BDRD

     0.9x1.25_gx1v6  B_1850_CN_CHEM 
     0.9x1.25_gx1v6  B_1850-2000_CN_CHEM

     1.9x2.5_gx1v6   B_1850_WACCM_CN
     1.9x2.5_gx1v6   B_1850-2000_WACCM_CN
  
     T31_gx3v7       B_1850_CN

  If the user is interested in running a "stand-alone" component configuration, 
  the following model configurations have been validated scientifically and 
  have associated diagnostic output as part of the release:

     1.9x2.5_1.9x2.5    F_2000_WACCM
     1.9x2.5_1.9x2.5    F_AMIP_CAM5
     1.9x2.5_1.9x2.5    F_AMIP_CN
     0.9x1.25_0.9x1.25  F_AMIP_CN

     0.9x1.25_gx1v6  I_2000
     0.9x1.25_gx1v6  I_2000_CN

     T62_gx1v6       C_NORMAL_YEAR

  For more information regarding alternative component configurations, 
  please refer to the individual component web pages at
  http://www.cesm.ucar.edu/models/cesm1.0

---

 CESM1 RESOLUTIONS:  name (shortname) 
   pt1_pt1 (pt1)  
   0.23x0.31_0.23x0.31 (f02_f02)  
   0.23x0.31_gx1v6 (f02_g16)  
   0.23x0.31_tx0.1v2 (f02_t12)  
   0.47x0.63_0.47x0.63 (f05_f05)  
   0.47x0.63_gx1v6 (f05_g16)  
   0.47x0.63_tx0.1v2 (f05_t12)  
   0.9x1.25_0.9x1.25 (f09_f09)  
   0.9x1.25_gx1v6 (f09_g16)  
   1.9x2.5_1.9x2.5 (f19_f19)  
   1.9x2.5_gx1v6 (f19_g16)  
   4x5_4x5 (f45_f45)  
   4x5_gx3v7 (f45_g37)  
   T62_gx3v7 (T62_g37)  
   T62_tx0.1v2 (T62_t12)  
   T62_gx1v6 (T62_g16)  
   T31_T31 (T31_T31)  
   T31_gx3v7 (T31_g37)  
   T42_T42 (T42_T42)  
   10x15_10x15 (f10_f10)  
   ne30np4_1.9x2.5_gx1v6 (ne30_f19_g16)  
   ne240np4_0.23x0.31_gx1v6 (ne240_f02_g16)  
   T85_T85 (T85_T85)  
 
 COMPSETS:  name (shortname): description 
   A_PRESENT_DAY (A) 
        Description: All data model  
   A_GLC (AG) 
        Description: All data model plus glc (glacier model)  
   B_2000 (B) 
        Description: All active components, present day  
   B_2000_CN (BCN) 
        Description: all active components, present day, with CN (Carbon Nitrogen) in clm  
   B_1850_CAM5 (B1850C5) 
        Description: All active components, pre-industrial, cam5 physics  
   B_1850 (B1850) 
        Description: All active components, pre-industrial  
   B_1850_CN (B1850CN) 
        Description: all active components, pre-industrial, with CN (Carbon Nitrogen) in CLM  
   B_2000_CN_CHEM (B2000CNCHM) 
        Description: All active components, pre-industrial, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm  
   B_1850_CN_CHEM (B1850CNCHM) 
        Description: All active components, pre-industrial, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm  
   B_1850_RAMPCO2_CN (B1850RMCN) 
        Description: All active components, pre-industirial with co2 ramp, with CN (Carbon Nitrogen) in CLM  
   B_1850-2000 (B20TR) 
        Description: All active components, 1850 to 2000 transient  
   B_1850-2000_CN (B20TRCN) 
        Description: All active components, 1850 to 2000 transient, with CN (Carbon Nitrogen) in CLM  
   B_1850-2000_CN_CHEM (B20TRCNCHM) 
        Description: All active components, 1850 to 2000 transient, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm  
   B_1850-2000_CAM5 (B20TRC5) 
        Description: All active components, 1850 to 2000 transient, cam5 physics  
   B_2000_GLC (BG) 
        Description: all active components, with active glc  
   B_2000_TROP_MOZART (BMOZ) 
        Description: All active components, with trop_mozart  
   B_1850_WACCM (B1850W) 
        Description: all active components, pre-industrial, with waccm  
   B_1850_WACCM_CN (B1850WCN) 
        Description: all active components, pre-industrial, with waccm and CN  
   B_1850-2000_WACCM_CN (B20TRWCN) 
        Description: All active components, 1850 to 2000 transient, WACCM with CN (Carbon Nitrogen) in CLM  
   B_1955-2005_WACCM_CN (B55TRWCN) 
        Description: All active components, 1955 to 2000 transient, WACCM with  daily solar data and SPEs, CLM with CN  
   B_1850_BGC-BPRP (B1850BPRP) 
        Description: All active components, pre-industrial, CN in CLM, ECO in POP, BGC CO2=prog, rad CO2=prog  
   B_1850_BGC-BDRD (B1850BDRD) 
        Description: All active components, pre-industrial, CN in CLM, ECO in POP, BGC CO2=diag, rad CO2=diag  
   B_1850-2000_BGC-BPRP (B20TRBPRP) 
        Description: All active components, 1850 to 2000 transient, CN in CLM, ECO in POP, BGC CO2=prog, rad CO2=prog  
   B_1850-2000_BGC-BDRD (B20TRBDRD) 
        Description: All active components, 1850 to 2000 transient, CN in CLM, ECO in POP, BGC CO2=diag, rad CO2=diag  
   C_NORMAL_YEAR_ECOSYS (CECO) 
        Description: Active ocean model with ecosys and with COREv2 normal year forcing  
   C_NORMAL_YEAR (C) 
        Description: Active ocean model with COREv2 normal year forcing  
   D_NORMAL_YEAR (D) 
        Description: Active ice model with COREv2 normal year forcing  
   E_2000 (E) 
        Description: Fully active cam and ice with som ocean, present day  
   E_2000_GLC (EG) 
        Description: Fully active cam and ice with som ocean and glc, present day  
   E_1850_CN (E1850CN) 
        Description: Pre-industrial fully active ice and som ocean, with CN  
   E_1850_CAM5 (E1850C5) 
        Description: Pre-industrial fully active ice and som ocean, cam5 physics   
   F_AMIP_CN (FAMIPCN) 
        Description: AMIP run for CMIP5 protocol - valid only for 1 degree cam/clm/pres-cice  
   F_AMIP_CAM5 (FAMIPC5) 
        Description: AMIP run for CMIP5 protocol with cam5  
   F_1850 (F1850) 
        Description: Pre-industrial cam/clm with prescribed ice/ocn  
   F_1850_CAM5 (F1850C5) 
        Description: Pre-industrial cam/clm with prescribed ice/ocn, cam5 physics  
   F_2000 (F) 
        Description: Stand-alone cam default, prescribed ocn/ice  
   F_2000_CAM5 (FC5) 
        Description: Stand-alone cam default, prescribed ocn/ice, cam5 physics  
   F_2000_CN (FCN) 
        Description: Stand-alone cam default, prescribed ocn/ice with CN  
   F_1850-2000_CN (F20TRCN) 
        Description: 20th Century transient stand-alone cam default, prescribed ocn/ice, with CN  
   F_2000_GLC (FG) 
        Description: Stand-alone cam default, prescribed ocn/ice, glc (glacier model)  
   F_1850_CN_CHEM (F1850CNCHM) 
        Description: stand-alone cam/clm, pre-industrial, with CN in CLM, super_fast_llnl chem in cam  
   F_1850_WACCM (F1850W) 
        Description: Pre-industrial cam/clm with prescribed ice/ocn  
   F_2000_WACCM (FW) 
        Description: present-day cam/clm with prescribed ice/ocn  
   G_1850_ECOSYS (G1850ECO) 
        Description: 1850 control for pop-ecosystem/cice/datm7/dlnd-rx1  
   G_NORMAL_YEAR (G) 
        Description: Coupled ocean ice with COREv2 normal year forcing  
   H_PRESENT_DAY (H) 
        Description: Coupled ocean ice slnd  
   I_2000 (I) 
        Description: Active land model with QIAN atm input data for 2003 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000  
   I_1850 (I1850) 
        Description: Active land model with QIAN atm input data for 1948 to 1972 and Satellite phenology (SP), CO2 level and Aerosol deposition for 1850  
   I_2000_GLC (IG) 
        Description: Active glacier model and active land model with QIAN atm input data for 2003 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000  
   I_1948-2004 (I4804) 
        Description: Active land model with QIAN atm input data for 1948 to 2004 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000  
   I_1850-2000 (I8520) 
        Description: Active land model with QIAN atm input data for 1948 to 2004 and transient Satellite phenology (SP), and Aerosol deposition from 1850 to 2000 and 2000 CO2 level  
   I_2000_CN (ICN) 
        Description: Active land model with QIAN atm input data for 2003 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 2000  
   I_1850_CN (I1850CN) 
        Description: Active land model with QIAN atm input data for 1948 to 1972 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 1850  
   I_1948-2004_CN (I4804CN) 
        Description: Active land model with QIAN atm input data for 1948 to 2004 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 2000  
   I_1850-2000_CN (I8520CN) 
        Description: Active land model with QIAN atm input data for 1948 to 1972 and transient CN, Aerosol dep from 1850 to 2000 and 2000 CO2 level  
   S_PRESENT_DAY (S) 
        Description: All stub models plus xatm  
   X_PRESENT_DAY (X) 
        Description: All dead model  
   XG_PRESENT_DAY (XG) 
        Description: All dead model and cism  
 
 MACHINES:  name (description)
   tcs (U of T IBM p6, os is AIX, 32 pes/node, batch system is Moab/LoadLeveler) 
   gpc (U of T iDataPlex intel cluster, os is linux, 8 pes/node, batch system is Moab/Torque) 
   bluefire (NCAR IBM p6, os is AIX, 32 pes/node, batch system is LSF) 
   brutus_po (Brutus Linux Cluster ETH (pgi/9.0-1 with open_mpi/1.4.1), 16 pes/node, batch system LSF, added by UB) 
   brutus_pm (Brutus Linux Cluster ETH (pgi/9.0-1 with mvapich2/1.4rc2), 16 pes/node, batch system LSF, added by UB) 
   brutus_io (Brutus Linux Cluster ETH (intel/10.1.018 with open_mpi/1.4.1), 16 pes/node, batch system LSF, added by UB) 
   brutus_im (Brutus Linux Cluster ETH (intel/10.1.018 with mvapich2/1.4rc2), 16 pes/node, batch system LSF, added by UB) 
   edinburgh_lahey (NCAR CGD Linux Cluster (lahey), 8 pes/node, batch system is PBS) 
   edinburgh_pgi (NCAR CGD Linux Cluster (pgi), 8 pes/node, batch system is PBS) 
   edinburgh_intel (NCAR CGD Linux Cluster (intel), 8 pes/node, batch system is PBS) 
   franklin (NERSC XT4, os is CNL, 4 pes/node, batch system is PBS) 
   hadley (UCB Linux Cluster, os is Linux (ia64), batch system is PBS) 
   hopper (NERSC XT5, os is CNL, 8 pes/node, batch system is PBS) 
   intrepid (ANL IBM BG/P, os is BGP, 4 pes/node, batch system is cobalt) 
   jaguar (ORNL XT4, os is CNL, 4 pes/node, batch system is PBS) 
   jaguarpf (ORNL XT5, os is CNL, 12 pes/node, batch system is PBS) 
   kraken (NICS/UT/teragrid XT5, os is CNL, 12 pes/node) 
   lynx_pgi (NCAR XT5, os is CNL, 12 pes/node, batch system is PBS) 
   midnight (ARSC Sun Cluster, os is Linux (pgi), batch system is PBS) 
   pleiades (NASA/AMES Linux Cluster, Linux (ia64), Altix ICE, 3.0 GHz Harpertown processors, 8 pes/node and 8 GB of memory, batch system is PBS) 
   pleiades_wes (NASA/AMES Linux Cluster, Linux (ia64), Altix ICE, 2.93 GHz Westmere processors, 12 pes/node and 24 GB of memory, batch system is PBS) 
   prototype_atlas (LLNL Linux Cluster, Linux (pgi), 8 pes/node, batch system is Moab) 
   prototype_hera (LLNL Linux Cluster, Linux (pgi), 16 pes/node, batch system is Moab) 
   prototype_columbia (NASA Ames Linux Cluster, Linux (ia64), 2 pes/node, batch system is PBS) 
   prototype_frost (NCAR IBM BG/L, os is BGL, 8 pes/node, batch system is cobalt) 
   prototype_nyblue (SUNY IBM BG/L, os is BGL, 8 pes/node, batch system is cobalt) 
   prototype_ranger (TACC Linux Cluster, Linux (pgi), 1 pes/node, batch system is SGE) 
   prototype_ubgl (LLNL IBM BG/L, os is BGL, 2 pes/node, batch system is Moab) 
   generic_ibm (generic ibm power system, os is AIX, batch system is LoadLeveler, user-defined) 
   generic_xt (generic CRAY XT, os is CNL, batch system is PBS, user-defined) 
   generic_linux_pgi (generic linux (pgi), os is Linux, batch system is PBS, user-defined) 
   generic_linux_lahey (generic linux (lahey), os is Linux, batch system is PBS, user-defined) 
   generic_linux_intel (generic linux (intel), os is Linux, batch system is PBS, user-defined) 
   generic_linux_pathscale (generic linux (pathscale), os is Linux, batch system is PBS, user-defined) 

---

Initializing the Model Setup:

The initial setup of the model on TCS is simplified with the short script below

#!/bin/bash

export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/$USER
export MACH=tcs
export COMPSET=B_1850_CN
export RES=f19_g16
export CASEROOT=~/runs/ccsm4_comp-${COMPSET}_res-${RES}

cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES -scratchroot $SCRATCH 

NOTE: $CCSMROOT should point to the model code version in /project/ccsm with the "_current" after it. The same for CESM1

NOTE: $CASEROOT which will specify your model run naming conventions will not archive properly (to short term archiving directory i.e. /scratch) if your CASEROOT name is too long. It is best to keep it short.

 

This script creates an 1850 control with all components of the model fully active and carbon nitrogen cycling in the land component, The resolution is 1.9x2.5 in the atmosphere and x1 in the ocean. The file is created in the ~/run directory:

For valid component sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/a2967.html For information on resolution sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/x42.html#ccsm_grids

Load Balancing:

For the NCAR bluefire load balancing table for a select set of simulations see: CESM1: http://www.cesm.ucar.edu/models/cesm1.0/timing/ CCSM4: http://www.cesm.ucar.edu/models/ccsm4.0/timing/

cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16

edit env_mach_pes.xml

<entry id="NTASKS_ATM"   value="448"  />   
<entry id="NTHRDS_ATM"   value="1"  />   
<entry id="ROOTPE_ATM"   value="0"  />   

<entry id="NTASKS_LND"   value="320"  />   
<entry id="NTHRDS_LND"   value="1"  />   
<entry id="ROOTPE_LND"   value="160"  />   

<entry id="NTASKS_ICE"   value="64"  />   
<entry id="NTHRDS_ICE"   value="1"  />   
<entry id="ROOTPE_ICE"   value="0"  />   

<entry id="NTASKS_OCN"   value="256"  />   
<entry id="NTHRDS_OCN"   value="1"  />   
<entry id="ROOTPE_OCN"   value="224"  />   

<entry id="NTASKS_CPL"   value="224"  />   
<entry id="NTHRDS_CPL"   value="1"  />   
<entry id="ROOTPE_CPL"   value="0"  />   

<entry id="NTASKS_GLC"   value="1"  />   
<entry id="NTHRDS_GLC"   value="1"  />   
<entry id="ROOTPE_GLC"   value="0"  />   

<entry id="PSTRID_ATM"   value="1"  />   
<entry id="PSTRID_LND"   value="1"  />   
<entry id="PSTRID_ICE"   value="1"  />   
<entry id="PSTRID_OCN"   value="1"  />   
<entry id="PSTRID_CPL"   value="1"  />   
<entry id="PSTRID_GLC"   value="1"  />   

Once this file is modified you can configure the case

./configure -case

You will notice that configure will change the file the you just edited and you can see the total processors used by the simulation (704 or 11 nodes in this case):

<entry id="TOTALPES"   value="704"  />   
<entry id="PES_LEVEL"   value="1r"  />   
<entry id="MAX_TASKS_PER_NODE"   value="64"  />   
<entry id="PES_PER_NODE"   value="64"  />   
<entry id="CCSM_PCOST"   value="-3"  />   
<entry id="CCSM_TCOST"   value="0"  />   
<entry id="CCSM_ESTCOST"   value="-3"  />   

Note: Rather than modifying the load balancing manually, NCAR has written a script that resides in your $CASE running directory that allows you to modify the individual component CPU allocation without playing with the env_mach_pes.xml file:

To try a different configuration we might want 8 cpus running the OCN component continually and the remaining 24 cpus running atm on 24 then LND, ICE and CPL on 8 each. To set this up you enter;

configure -cleanmach
xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val 24
xmlchange -file env_mach_pes.xml -id NTASKS_LND -val 8 
xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val 8
xmlchange -file env_mach_pes.xml -id ROOTPE_ICE -val 8
xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val 8
xmlchange -file env_mach_pes.xml -id ROOTPE_CPL -val 16
xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val 8
xmlchange -file env_mach_pes.xml -id ROOTPE_OCN -val 24
configure -case

Then build and resubmit

The task geometry used by loadleveler on TCS is located in the file: ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run

Ensure that the proper modules are loaded:

Currently Loaded Modulefiles:

 1) ncl/5.1.1               3) netcdf/4.0.1_nc3        5) xlf/13.1
 2) nco/3.9.6               4) parallel-netcdf/1.1.1   6) vacpp/11.1

Now compile the model with:

./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build

One of the pre-processing steps in this build sequence is to fetch inputdat sets (initial and boundary conditions) from the NCAR SVN server. You may want to do this yourself before you build on the datamover1 node if there is a large amount of initial condition data to transfer from the NCAR repository. datamover1 has a high bandwidth connection to the outside. Note: We have most of the input data on /project/ccsm already so this step will not be required for the more common configurations.

> ssh datamover1
Last login: Wed Jul  7 16:38:14 2010 from tcs-f11n06-gpfs
user@gpc-logindm01:~>cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>./check_input_data -inputdata /project/ccsm/inputdata -export
Input Data List Files Found:
./Buildconf/cam.input_data_list
./Buildconf/clm.input_data_list
./Buildconf/cice.input_data_list
./Buildconf/pop2.input_data_list
./Buildconf/cpl.input_data_list
export https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc /project/ccsm/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc ..... success

Setting the Simulation Length:

The amount of time that you would like to run the model can be set by editing env_run.xml at anytime in the setup sequence

<entry id="RESUBMIT"   value="10"  />   

<entry id="STOP_OPTION"   value="nmonths"  />   

<entry id="STOP_N"   value="12"  />   

These settings will tell the model to checkpoint after each model year (12 months) and run for a total of 10 years (10 checkpoints)

Running CCSM4 on the Distributed System (TCS):

The model is now ready to be submitted to the TCS batch queue

llsubmit ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run

Once the model has run through a checkpoint timing information on the simulation will be found in:

~/runs/ccsm4_comp-B_1850_CN_res-f19_g16/timing

Standard output from the model can be followed during runtime by going to:

/scratch/guido/ccsm4_comp-B_1850_CN_res-f19_g16/run

and running

tail -f <component_log_file>

The model will archive the NetCDF output in:

/scratch/$USER/archive

Cloning Simulations

A useful command that allow for the setup of multiple runs quickly is the clone command. It allows for the cloning of a case quickly (so there is no need to run the setup script above every time)

cd ~/runs
/project/ccsm/ccsm4_0_current/scripts/create_clone -clone ccsm4_comp-B_1850_CN_res-f09_g16 -case ccsm4_comp-B_1850_CN_res-f09_g16_clone -v

To change the load balancing (env_mach_pes.xml) in a current simulation setup or other parameters you can do a clean build to make sure the model is rebuilt properly:

./configure -cleanmach
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.clean_build
./configure -case
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build

Running CCSM4 on GPC

The setup script is almost identical:

#!/bin/bash

export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/guido
export MACH=gpc
export COMPSET=B_1850_CN
export RES=f09_g16
export CASEROOT=~/runs/ccsm4gpc_comp-${COMPSET}_res-${RES}

cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES 

To load balance and run the model follow the steps above: The env_mach_pes.xml configuration files needs to be modified as follows:

<entry id="MAX_TASKS_PER_NODE"   value="8"  />   
<entry id="PES_PER_NODE"   value="8"  />   

Use qsub to submit the model to the GPC cluster:

qsub ccsm4gpc_comp-B_1850_CN_res-f19_g16.tcs.run

 

Running CCSM4 interactively on TCS

In the run script you need to specify the following set and unset of environment variables. You also need a hostfile that will describe the node procs that you will be using

unsetenv MP_EUILIB

setenv MP_PROCS 16 

setenv MP_NODES 1
  

/usr/bin/poe /project/ccsm/bin/ccsm_launch ./ccsm.exe -hfile /project/<user>/runs/<casename>/hostfile

and the hostfile looks like (for 16 threads of tcs02)

--> more hostfile
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06