Run

As in Build, we are on Derecho.

Global¶

We will use the quasi-uniform 240-km mesh (10242 cells), as in the official virtual tutorial (Section 1.3).

Run directory¶

First we build out our run directory.

cd $SCRATCH
mkdir mpas-global-240km
cd $_

ln -s /glade/campaign/mmm/wmr/mpas_tutorial/meshes/x1.10242.grid.nc .

ln -s /glade/campaign/mmm/wmr/mpas_tutorial/meshes/x1.10242.graph.info.part.32 .
ln -s ~/MPAS-Model_v8.3/init_atmosphere_model .
cp ~/MPAS-Model_v8.3/namelist.init_atmosphere .
cp ~/MPAS-Model_v8.3/streams.init_atmosphere .

ln -s /glade/campaign/mmm/wmr/mpas_tutorial/meshes/x1.10242.graph.info.part.128 .
ln -s ~/MPAS-Model_v8.3/atmosphere_model .
cp ~/MPAS-Model_v8.3/namelist.atmosphere .
cp ~/MPAS-Model_v8.3/streams.atmosphere .
cp ~/MPAS-Model_v8.3/stream_list.atmosphere.* .
ln -s ~/MPAS-Model_v8.3/src/core_atmosphere/physics/physics_wrf/files/* .

Static file¶

Now we edit the init_atmosphere config files to set up for generating the static file.

Update namelist.init_atmosphere with these settings:

Then, in streams.init_atmosphere, set the input file name template to x1.10242.grid.nc, and the output file name template to x1.10242.static.nc.

Create a job script to run the model initialization program.

#!/usr/bin/env bash
# Based on /glade/campaign/mmm/wmr/mpas_tutorial/job_scripts/init_real.pbs

#
# For more information on submitting jobs to Derecho, see this documentation:
# https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/job-scripts/
#

#--- Give our job a reasonable name
#PBS -N init_real

#--- Run in the main queue
#PBS -q main
#PBS -l job_priority=regular

#--- Set the project under which the job will run
#PBS -A UTAM0025

#--- Specify wallclock limit and resources
#PBS -l walltime=00:30:00
#PBS -l select=1:ncpus=128:mpiprocs=32:mem=235gb

source ~/mpas-modules-intel.sh

mpiexec ./init_atmosphere_model

Submit the job.

qsub init_real.pbs

qstat -u $USER

watch -n1 qstat -u $USER

qstat -u $USER -f -w

cp /glade/derecho/scratch/knocasio/mpas-global240km/x1.10242.static.nc .

Initial conditions¶

20 September 2017, 00 UTC, the day when Hurricane Maria made landfall in Puerto Rico.

First we need WPS intermediate files.

The era5_to_int tool can be used to create intermediate files from the NSF NCAR RDA ERA5 netCDF files on GLADE. But here, to simplify things, we will use an intermediate file that we have already created.

ln -s /glade/u/home/zmoon/mpas-tutorial/global/FILE:2017-09-20_00 .

/glade/u/home/wrfhelp/derecho_pre_compiled_code/wpsv4.5/util/rd_intermediate.exe

Update namelist.init_atmosphere with these settings:

Then, in streams.init_atmosphere, set the input file name template to x1.10242.static.nc (the static file we just created), and the output file name template to x1.10242.init.nc.

Submit the job (same one we used to create the static file):

qsub init_real.pbs

cp /glade/derecho/scratch/knocasio/mpas-global240km/x1.10242.init.nc .

Run the model¶

Update namelist.atmosphere with these settings:

Then, in streams.atmosphere, set the input file name template to x1.10242.init.nc (the initial conditions file we just created).

Create a job script to run the model.

#!/usr/bin/env bash
# Based on /glade/campaign/mmm/wmr/mpas_tutorial/job_scripts/run_model.pbs

#
# For more information on submitting jobs to Derecho, see this documentation:
# https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/job-scripts/
#

#--- Give our job a reasonable name
#PBS -N run_model

#--- Run in the main queue
#PBS -q main
#PBS -l job_priority=regular

#--- Set the project under which the job will run
#PBS -A UTAM0025

#--- Specify wallclock limit and resources
#PBS -l walltime=00:30:00
#PBS -l select=1:ncpus=128:mpiprocs=128:mem=235gb

source ~/mpas-modules-intel.sh

mpiexec ./atmosphere_model

Submit the job.

qsub run_model.pbs

The main model log is log.atmosphere.0000.out. To follow the updates:

tail -f log.atmosphere.0000.out

A successful run will produce various model output files:

• diag.*.nc (mostly 2-d diagnostic fields; specified in stream_list.atmosphere.diagnostics)
• history.*.nc (2- and 3-d prognostic and diagnostic fields; specified in stream_list.atmosphere.output)
• restart.*.nc (checkpoints of the model state that we can use for a warm start)

Regional¶

We will use the limited-area domain as in:

• Núñez Ocasio et al. (2024)
• Núñez Ocasio & Dougherty (2024)
• Núñez Ocasio et al. (2025)
• Núñez Ocasio et al. (2025)

created by rotating, moving, and cropping the x5.8060930 global 15–3-km elliptical refinement mesh. It has 6092549 cells (~6M; ~600x the global example).

Run directory¶

First we need to generate the tables for the Thompson microphysics scheme. As when we built the model, we can use an interactive job.

qsub -I -l walltime=3600 -l select=1:ncpus=2:mem=16gb -A UTAM0025 -q develop

source ~/mpas-modules-intel.sh

cd ~/MPAS-Model_v8.3

./build_tables

mv MP_THOMPSON_* src/core_atmosphere/physics/physics_wrf/files/
ln -s src/core_atmosphere/physics/physics_wrf/files/MP_THOMPSON_* .

exit

As before, we first build out our run directory. We use the same MPAS-Model directory, but copy a different mesh.

cd $SCRATCH
mkdir mpas-africa
cd $_

ln -s /glade/u/home/zmoon/mpas-tutorial/africa/Africa.static.nc .

ln -s /glade/u/home/zmoon/mpas-tutorial/africa/Africa.graph.info.part.240 .
ln -s ~/MPAS-Model_v8.3/init_atmosphere_model .
cp ~/MPAS-Model_v8.3/namelist.init_atmosphere .
cp ~/MPAS-Model_v8.3/streams.init_atmosphere .

ln -s /glade/u/home/zmoon/mpas-tutorial/africa/Africa.graph.info.part.5400 .
ln -s ~/MPAS-Model_v8.3/atmosphere_model .
cp ~/MPAS-Model_v8.3/namelist.atmosphere .
cp ~/MPAS-Model_v8.3/streams.atmosphere .
cp ~/MPAS-Model_v8.3/stream_list.atmosphere.* .
ln -s ~/MPAS-Model_v8.3/src/core_atmosphere/physics/physics_wrf/files/* .

Note that we link a regional static file instead of a grid file (we’re skipping static file creation in this example). See Section 5.1 of the official virtual tutorial for some guidance on creating regional static files.

Initial conditions¶

12 September 2017, 00 UTC, the day when the AEW that would become Hurricane Maria left the African coast.

As in the global example, we will link the needed WPS intermediate file (and those we need for the LBCs as well).

ln -s /glade/derecho/scratch/zmoon/mpas-africa/FILE:* .

Update namelist.init_atmosphere with these settings:

Then, in streams.init_atmosphere, set the input file name template to Africa.static.nc and the output file name template to Africa.init.nc.

Create a job script to run the model initialization program.

#!/usr/bin/env bash
#PBS -N init
#PBS -q main
#PBS -l job_priority=regular
#PBS -A UTAM0025
#PBS -l walltime=01:30:00
#PBS -l select=2:ncpus=128:mpiprocs=120:mem=235gb

source ~/mpas-modules-intel.sh

mpiexec ./init_atmosphere_model

Submit the job.

qsub init.pbs

ln -s /glade/derecho/scratch/knocasio/mpas-africa/Africa.init.nc .

Boundary conditions¶

Update namelist.init_atmosphere with these settings:

👆 Compared to Table 4, the differences are that now we are using init case 9, and we need to set a stop time.

Then, in streams.init_atmosphere, set the input file name template to Africa.init.nc, the output file name template to foo.nc (unused, but it can’t be the same as others), and the LBC output interval to 1:00:00 (hourly; consistent with our config_fg_interval).

Submit the job (same one we used to create the initial conditions):

qsub init.pbs

We will again skip creating surface update files.

ln -s /glade/derecho/scratch/knocasio/mpas-africa/lbc.*.nc .

Run the model¶

Update namelist.atmosphere with these settings:

👆 Note that we have set a much smaller time step than in the coarse global example, we turn on application of LBCs, we use the default RT interval, and we have selected the convection-permitting physics suite.

Then, in streams.atmosphere

• set the input file name template to Africa.init.nc
• set the restart output interval to 1_12:00:00 (the end of our run)
• set the diagnostics output interval to 1:00:00 (hourly)
• set the LBC input interval to 1:00:00 to match our LBC files

Finally, in stream_list.atmosphere.diagnostics, replace the contents with

initial_time
xtime
Time
olrtoa
rainc
rainnc
t_isobaric
uzonal_isobaric
umeridional_isobaric
vorticity_isobaric

👆 In the output diag files, the *_isobaric variables will be on pressure levels, specifically the 27 ERA5 levels from 100 to 1000 hPa.

Create a job script to run the model.

#!/usr/bin/env bash
#PBS -N run
#PBS -q main
#PBS -l job_priority=regular
#PBS -A UTAM0025
#PBS -l walltime=12:00:00
#PBS -l select=45:ncpus=128:mpiprocs=120:mem=235gb

source ~/mpas-modules-intel.sh

mpiexec ./atmosphere_model

👆 Note that the product of the number of nodes (select) and the number of MPI processes per node (mpiprocs) matches our second partition file.

Submit the job.

qsub run.pbs