本文主要参考https://www.myroms.org/wiki/ROMS_UNSW2008
本文所使用的ROMS版本为(3.7,version963)
特别注意的是,在官网中https://www.myroms.org/wiki/Getting_Started 中有一行很不起眼的小字,是说2019年1月的一次更新中很多文件名发生了变化,而官网中大部分入门教程并未更新,就会导致你照做会不断报错。因此,在踩了很多个坑以后,有了这篇文章。
NoteNote: In SVN revision 933 (January 26, 2019) all ocean_.in files were renamed to roms_.in to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository Trac ticket #794. If you are working with a ROMS release prior to revision 933 you will need to replace roms_upwelling.in with ocean_upwelling.in in all commands below.
======================================================================================
具体区别在于:
reveision 933 以前的版本:
ocean_*.in,ocean_*.nc,./oceanM,./oceanS,./oceanG,build.bash,build文件夹
reveision 933 以后的版本:
roms_*.in ,roms_*.nc ,./romsM,./romsS,./romsG,build_roms.bash,build_roms文件夹
超级坑!!!
======================================================================================
1.安装
# 清除已有module
module purge
# netCDF
# module load netCDF/4.4.1-intel-2016b
module load netCDF-Fortran/4.4.4-intel-2016b
module load flex/2.6.2-intel-2016b
export NETCDF=$EBROOTNETCDFMINFORTRAN
export NETCDF_INCDIR=$EBROOTNETCDFMINFORTRAN/include
export NETCDF_LIBDIR=$EBROOTNETCDFMINFORTRAN/lib
# 加载完module,你将得到以下全部module
[hengkai.yao@ada2 Projects]$ module list
Currently Loaded Modules:
1) GCCcore/5.4.0 10) zlib/1.2.8-intel-2016b
2) binutils/2.26-GCCcore-5.4.0 11) HDF5/1.8.17-intel-2016b
3) icc/2016.3.210-GCC-5.4.0-2.26 12) cURL/7.49.1-intel-2016b
4) ifort/2016.3.210-GCC-5.4.0-2.26 13) Szip/2.1-intel-2016b
5) iccifort/2016.3.210-GCC-5.4.0-2.26 14) netCDF/4.4.1-intel-2016b
6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26 15) netCDF-Fortran/4.4.4-intel-2016b
7) iimpi/2016b 16) M4/1.4.17-intel-2016b
8) imkl/11.3.3.210-iimpi-2016b 17) flex/2.6.2-intel-2016b
9) intel/2016b
通过单独load
每一个模块可以看到所加载的具体依赖项,同时发现netcdf-fortran
包括了netcdf
,因此没必要多花7s去load netcdf
。
[hengkai.yao@ada2 Projects]$ module load netCDF/4.4.1-intel-2016b
Currently Loaded Modules:
1) GCCcore/5.4.0 8) imkl/11.3.3.210-iimpi-2016b
2) binutils/2.26-GCCcore-5.4.0 9) intel/2016b
3) icc/2016.3.210-GCC-5.4.0-2.26 10) zlib/1.2.8-intel-2016b
4) ifort/2016.3.210-GCC-5.4.0-2.26 11) HDF5/1.8.17-intel-2016b
5) iccifort/2016.3.210-GCC-5.4.0-2.26 12) cURL/7.49.1-intel-2016b
6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26 13) Szip/2.1-intel-2016b
7) iimpi/2016b 14) netCDF/4.4.1-intel-2016b
[hengkai.yao@ada2 Projects]$ module load netCDF-Fortran/4.4.4-intel-2016b
Currently Loaded Modules:
1) GCCcore/5.4.0 9) intel/2016b
2) binutils/2.26-GCCcore-5.4.0 10) zlib/1.2.8-intel-2016b
3) icc/2016.3.210--5.4.0-2.26 11) HDF5/1.8.17-intel-2016b
4) ifort/2016.3.210-GCC-5.4.0-2.26 12) cURL/7.49.1-intel-2016b
5) iccifort/2016.3.210-GCC-5.4.0-2.26 13) Szip/2.1-intel-2016b
6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26 14) netCDF/4.4.1-intel-2016b
7) iimpi/2016b 15) netCDF-Fortran/4.4.4-intel-2016b
8) imkl/11.3.3.210-iimpi-2016b
[hengkai.yao@ada2 Projects]$ module load flex/2.6.2-intel-2016b
Currently Loaded Modules:
1) GCCcore/5.4.0 7) iimpi/2016b
2) binutils/2.26-GCCcore-5.4.0 8) imkl/11.3.3.210-iimpi-2016b
3) icc/2016.3.210-GCC-5.4.0-2.26 9) intel/2016b
4) ifort/2016.3.210-GCC-5.4.0-2.26 10) M4/1.4.17-intel-2016b
5) iccifort/2016.3.210-GCC-5.4.0-2.26 11) flex/2.6.2-intel-2016b
6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26
2.下载最新版ROMS
# 创建新文件夹
[hengkai.yao@ada2 Projects]$ cd ~
[hengkai.yao@ada2 ~]$ mkdir ROMS
# 下载最新的ROMS版本
cd ROMS
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/trunk trunk
3.自定义build脚本
# 创建项目文件夹
[hengkai.yao@ada2 ROMS]$ mkdir Projects
[hengkai.yao@ada2 ROMS]$ cd Projects
# 创建upwelling项目
[hengkai.yao@ada2 Projects]$ mkdir Upwelling
[hengkai.yao@ada2 Projects]$ cd Upwelling
# 拷贝输入文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/External/roms_upwelling.in .
# 拷贝头文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Include/upwelling.h .
# 拷贝build脚本,注意这里文件名与旧版本的不同
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Bin/build_roms.bash .
# 在build_roms.bash文件内设置参数,根据实际文件所在位置填写
export MY_ROOT_DIR=/home/hengkai.yao/ROMS 或 MY_ROOT_DIR=${HOME}/ROMS 或 MY_ROOT_DIR=/scratch/user/hengkai.yao/ROMS
export MY_PROJECT_DIR=${MY_ROOT_DIR}/Projects/Upwelling
export MY_ROMS_SRC=${MY_ROOT_DIR}/trunk
# 编译器设置
#export MY_CPP_FLAGS="-DNPZD_POWELL"#需要被注释掉,不运行
# 考虑本例在集群上并行运算,开启以下开关,并使用ifort编译器
export USE_MPI=on
export USE_MPIF90=on
export FORT=ifort
export USE_NETCDF4=on
# 不使用自定义库
export USE_MY_LIBS=no
至此的文件结构如下:
[hengkai.yao@ada5 ~]$ tree -L 3
└── ROMS
├── Projects
│ └── Upwelling
└── trunk
├── Compilers
├── Data
├── ESM
├── Lib
├── makefile
├── Master
├── ROMS
├── User
└── Waves
4.编译
# 单核编译
[hengkai.yao@ada2 Projects]$ cd Upwelling
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash
# 多核编译
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash -j 8
5.运行
# 5.1单核运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 1 ./romsM roms_upwelling.in > my_upwelling.log & top
# 5.2多核运行
运行前需要先修改roms_upwelling.in文件中的核心设置
! Domain decomposition parameters for serial, distributed-memory or
! shared-memory configurations used to determine tile horizontal range
! indices (Istr,Iend) and (Jstr,Jend), [1:Ngrids].
NtileI == 2 ! I-direction partition
NtileJ == 2 ! J-direction partition
# 然后命令行运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 4 ./romsM roms_upwelling.in > my_upwelling.log & top
多核心运行后top能够看到正在运行的线程有4个来自romsM,一个来自top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9215 hengkai. 20 0 217m 45m 19m R 100.0 0.0 0:34.23 romsM
9217 hengkai. 20 0 216m 44m 19m R 100.0 0.0 0:34.47 romsM
9216 hengkai. 20 0 214m 41m 18m R 99.9 0.0 0:34.46 romsM
9218 hengkai. 20 0 215m 43m 19m R 99.9 0.0 0:34.47 romsM
9047 hengkai. 20 0 15576 1980 1008 R 0.7 0.0 0:00.29 top
6.成功运行完毕
当top线程监控中不再有romsM说明已运行完毕或运行当中出错,此时应打开tail my_upwelling.log
看一下结果。
若果出现一下内容(DONE
)即完成运行,同时在Upwelling文件夹中多出很多.nc
文件,特别是roms_rst.nc
即为输出的数据。
Analytical header files used:
ROMS/Functionals/ana_btflux.h
ROMS/Functionals/ana_grid.h
ROMS/Functionals/ana_initial.h
ROMS/Functionals/ana_smflux.h
ROMS/Functionals/ana_stflux.h
ROMS/Functionals/ana_vmix.h
ROMS/TOMS: DONE... Tuesday - April 9, 2019 - 5:58:48 PM
log文件头有这次实验的描述:
--------------------------------------------------------------------------------
Model Input Parameters: ROMS/TOMS version 3.7
Tuesday - April 9, 2019 - 10:31:24 PM
--------------------------------------------------------------------------------
Wind-Driven Upwelling/Downwelling over a Periodic Channel
Operating system : Linux
CPU/hardware : x86_64
Compiler system : ifort
Compiler command : /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/b
Compiler flags : -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit
MPI Communicator : 1140850688 PET size = 4
Input Script : roms_upwelling.in
SVN Root URL : https://www.myroms.org/svn/src/trunk
SVN Revision : 963M
Local Root : /scratch/user/hengkai.yao/ROMS/trunk
Header Dir : /scratch/user/hengkai.yao/ROMS/Projects/Upwelling
Header file : upwelling.h
Analytical Dir: /scratch/user/hengkai.yao/ROMS/Projects/Upwelling
Resolution, Grid 01: 41x80x16, Parallel Nodes: 4, Tiling: 2x2
7.报错
1.参数未固定路径
1.1报错形式
[hengkai.yao@ada6 Upwelling]$ ./build.bash ./build.bash: line 248: cd:
/Users/hengkai.yao/ROMS/trunk: No such file or directory make: *** No
rule to make target `clean'. Stop. make: *** No targets specified and
no makefile found. Stop.
1.2解决方案:参数改成相对路径
! Input ASCII parameter filenames.
APARNAM = ROMS/External/s4dvar.in
SPOSNAM = ROMS/External/stations.in
FPOSNAM = ROMS/External/floats.in
BPARNAM = ROMS/External/bio_Fennel.in
SPARNAM = ROMS/External/sediment.in
USRNAME = ROMS/External/MyFile.dat
2.ifort调用mpif90错误
原本应该用ifort内的mpiifort,而ROMS程序默认使用了gfortran家的mpif90。这是一个纯经验性错误,非常变态。
2.1报错形式
cd /home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms; /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/bin64/mpif90 -c -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit -free mod_kinds.f90
gfortran: error: precise: No such file or directory
gfortran: error: uninit: No such file or directory
gfortran: error: unrecognized command line option ‘-fp-model’
gfortran: error: unrecognized command line option ‘-h’
gfortran: error: unrecognized command line option ‘-ip’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-check’
make: *** [/home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms/mod_kinds.o] Error 1
2.2解决方案
# 改ROMS/trunk/Compilers/Linux-ifort.mk
ifdef USE_MPIF90
FC := mpif90
改成
ifdef USE_MPIF90
FC := mpiifort
3.运行报错
3.1报错形式
MOD_NCPARAM - Unable to open variable information file:
ROMS/External/varinfo.dat
Default file is located in source directory.
3.2报错说明
说明Upwelling文件夹内的varinfo.dat未被调用,调用的是ROMS/trunk内的varinfo.dat,因此需要指定roms_upwelling.in文件内的varinfo.dat路径。
3.3解决方案
打开/Upwelling文件夹内的roms_upwelling.in,修改如下项至pwd位置。
VARNAME = varinfo.dat
4.netcdf报错
mod_netcdf.f90:(.text+0x21894): undefined reference to `netcdf_mp_nf90_redef_'
/scratch/user/hengkai.yao/ROMS/test/upwelling/Build_roms/libMODS.a(mod_netcdf.o): In function `mod_netcdf_mp_netcdf_sync_':
mod_netcdf.f90:(.text+0x21af4): undefined reference to `netcdf_mp_nf90_sync_'
make: *** [/scratch/user/hengkai.yao/ROMS/test/upwelling/romsM] Error 1
我猜是 export USE_NETCDF4=on 没开
5.日常拷贝下载文件报错
disk quota exceeded
说明你的硬盘大小或文件数目限额到达,需转移文件到限额大的硬盘中。
TIP
在查bug的时候,无意中看到jcwarner大神在ROMS官方论坛的一段留言
if you set
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
or
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90=on # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
then you will get an oceanM
this will give you an oceanO
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP=on # shared-memory parallelism
if you do this:
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
you will get an oceanS.
画图
# 加载ROMS-matlab包
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/matlab
addpath('/home/hengkai.yao/ROMS/matlab','-end')
小技巧
[hengkai.yao@ada6 matlab]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 29G 10G 18G 37% /
tmpfs 127G 26M 127G 1% /dev/shm
/dev/sda2 240M 92M 136M 41% /boot
/dev/sda1 50M 254K 50M 1% /boot/efi
/dev/mapper/os-tmp 20G 424M 18G 3% /tmp
/dev/mapper/os-vartmp
9.6G 23M 9.0G 1% /var/tmp
scratch 1.9P 1.5P 325T 83% /scratch
tiered 923T 829T 95T 90% /tiered
general 121T 12T 109T 10% /general
[hengkai.yao@ada6 ~]$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/os-root 1921360 288903 1632457 16% /
tmpfs 33058969 2173 33056796 1% /dev/shm
/dev/sda2 65536 44 65492 1% /boot
/dev/sda1 0 0 0 - /boot/efi
/dev/mapper/os-tmp 1281120 2969 1278151 1% /tmp
/dev/mapper/os-vartmp
642112 20 642092 1% /var/tmp
scratch 315000576 204880316 110120260 66% /scratch
tiered 368435456 138822622 229612834 38% /tiered
general 325545984 41343125 284202859 13% /general
P.S.
推荐一个小命令sshpass,需要在本机安装,一键登录集群,非常方便。
Mac上的安装指令:
brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb
用法:
sshpass -p 'yourPassword' ssh yourUsername@yourServerIP
10 comments
感谢博主的文章 很受用 现在遇到了个问题 模式的编译没有问题 在运行的时候 终端只显示了TITLE (就是模式的版本号以及开始运行的时间,接下来就没有任何显示了。我想模式应该没有在运行,top的进程是有显示模式在运行 但就是没有结果nc文件生成 请问有什么办法吗
如果你是使用类似于nohup mpirun -np 1 ./romsM roms_upwelling.in > my_upwelling.log & top这样子运行的,nohup就是不显示在屏幕上, > my_upwelling.log就是输出给这个log文件,你可以tail 这个log文件看看有没有报错以及有没有运行完,如果没有报错说明模式刚开始运行还没到生成输出nc的步骤,如果报错了,说明模式停止了,也没运行到输出nc的那一步
天哪,谢谢大佬ヾ(≧∇≦*)ゝ
不是啥大佬,祝模式运行的顺利
这里为啥要用module,有什么作用吗
关于module系统,参阅https://hprc.tamu.edu/wiki/SW:Modules
module是超算系统中常用的一个包,用于统一管理各种编译包和软件,这样每个用户不必单独安装各个compiler,而是由管理员统一安装一次,每个用户在使用前使用module命令调用即可。这个方法在大用户量的集群中十分常见。
更改文件名真是坑
是呀,所以不妨试试旧版本
太棒了!!