ROMS:从入门到精通-1.Upwelling Case

Author： Eddy
发布时间：April 13, 2019
3701views
6 comments
13661 words
Categories： Academic

本文主要参考https://www.myroms.org/wiki/ROMS_UNSW2008
本文所使用的ROMS版本为（3.7,version963）
特别注意的是，在官网中https://www.myroms.org/wiki/Getting_Started 中有一行很不起眼的小字，是说2019年1月的一次更新中很多文件名发生了变化，而官网中大部分入门教程并未更新，就会导致你照做会不断报错。因此，在踩了很多个坑以后，有了这篇文章。

NoteNote: In SVN revision 933 (January 26, 2019) all ocean_.in files were renamed to roms_.in to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository Trac ticket #794. If you are working with a ROMS release prior to revision 933 you will need to replace roms_upwelling.in with ocean_upwelling.in in all commands below.

======================================================================================
具体区别在于：
reveision 933 以前的版本：
ocean_*.in，ocean_*.nc，./oceanM，./oceanS，./oceanG，build.bash，build文件夹
reveision 933 以后的版本：
roms_*.in ，roms_*.nc ，./romsM，./romsS，./romsG，build_roms.bash，build_roms文件夹
超级坑！！！
======================================================================================

1.安装

# 清除已有module
module purge
# netCDF
# module load netCDF/4.4.1-intel-2016b
module load netCDF-Fortran/4.4.4-intel-2016b
module load flex/2.6.2-intel-2016b
export NETCDF=$EBROOTNETCDFMINFORTRAN
export NETCDF_INCDIR=$EBROOTNETCDFMINFORTRAN/include
export NETCDF_LIBDIR=$EBROOTNETCDFMINFORTRAN/lib
# 加载完module，你将得到以下全部module
[hengkai.yao@ada2 Projects]$ module list
Currently Loaded Modules:
  1) GCCcore/5.4.0                                      10) zlib/1.2.8-intel-2016b
  2) binutils/2.26-GCCcore-5.4.0                        11) HDF5/1.8.17-intel-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      12) cURL/7.49.1-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    13) Szip/2.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 14) netCDF/4.4.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  15) netCDF-Fortran/4.4.4-intel-2016b
  7) iimpi/2016b                                        16) M4/1.4.17-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b                        17) flex/2.6.2-intel-2016b
  9) intel/2016b

通过单独load每一个模块可以看到所加载的具体依赖项，同时发现netcdf-fortran包括了netcdf，因此没必要多花7s去load netcdf。

[hengkai.yao@ada2 Projects]$ module load netCDF/4.4.1-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       8) imkl/11.3.3.210-iimpi-2016b
  2) binutils/2.26-GCCcore-5.4.0                         9) intel/2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      10) zlib/1.2.8-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    11) HDF5/1.8.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 12) cURL/7.49.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  13) Szip/2.1-intel-2016b
  7) iimpi/2016b                                        14) netCDF/4.4.1-intel-2016b

[hengkai.yao@ada2 Projects]$ module load netCDF-Fortran/4.4.4-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       9) intel/2016b
  2) binutils/2.26-GCCcore-5.4.0                        10) zlib/1.2.8-intel-2016b
  3) icc/2016.3.210--5.4.0-2.26                      11) HDF5/1.8.17-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    12) cURL/7.49.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 13) Szip/2.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  14) netCDF/4.4.1-intel-2016b
  7) iimpi/2016b                                        15) netCDF-Fortran/4.4.4-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b

[hengkai.yao@ada2 Projects]$ module load flex/2.6.2-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       7) iimpi/2016b
  2) binutils/2.26-GCCcore-5.4.0                         8) imkl/11.3.3.210-iimpi-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                       9) intel/2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    10) M4/1.4.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 11) flex/2.6.2-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26

2.下载最新版ROMS

#  创建新文件夹
[hengkai.yao@ada2 Projects]$ cd ~
[hengkai.yao@ada2 ~]$ mkdir ROMS
# 下载最新的ROMS版本
cd ROMS
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/trunk trunk

3.自定义build脚本

# 创建项目文件夹
[hengkai.yao@ada2 ROMS]$ mkdir Projects
[hengkai.yao@ada2 ROMS]$ cd Projects
# 创建upwelling项目
[hengkai.yao@ada2 Projects]$ mkdir Upwelling
[hengkai.yao@ada2 Projects]$ cd Upwelling
# 拷贝输入文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/External/roms_upwelling.in .
# 拷贝头文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Include/upwelling.h .
# 拷贝build脚本，注意这里文件名与旧版本的不同
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Bin/build_roms.bash .
# 在build_roms.bash文件内设置参数，根据实际文件所在位置填写
 export        MY_ROOT_DIR=/home/hengkai.yao/ROMS 或 MY_ROOT_DIR=${HOME}/ROMS 或 MY_ROOT_DIR=/scratch/user/hengkai.yao/ROMS
 export     MY_PROJECT_DIR=${MY_ROOT_DIR}/Projects/Upwelling
 export        MY_ROMS_SRC=${MY_ROOT_DIR}/trunk
# 编译器设置
#export       MY_CPP_FLAGS="-DNPZD_POWELL"#需要被注释掉，不运行
# 考虑本例在集群上并行运算，开启以下开关，并使用ifort编译器
 export            USE_MPI=on
 export         USE_MPIF90=on
 export               FORT=ifort
 export       USE_NETCDF4=on
# 不使用自定义库
export           USE_MY_LIBS=no

至此的文件结构如下：

[hengkai.yao@ada5 ~]$ tree -L 3

└── ROMS
    ├── Projects
    │   └── Upwelling
    └── trunk
        ├── Compilers
        ├── Data
        ├── ESM
        ├── Lib
        ├── makefile
        ├── Master
        ├── ROMS
        ├── User
        └── Waves

4.编译

# 单核编译
[hengkai.yao@ada2 Projects]$ cd Upwelling
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash
# 多核编译
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash -j 8

5.运行

# 5.1单核运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 1 ./romsM roms_upwelling.in > my_upwelling.log & top
# 5.2多核运行
运行前需要先修改roms_upwelling.in文件中的核心设置
! Domain decomposition parameters for serial, distributed-memory or
! shared-memory configurations used to determine tile horizontal range
! indices (Istr,Iend) and (Jstr,Jend), [1:Ngrids].

      NtileI == 2                               ! I-direction partition
      NtileJ == 2                               ! J-direction partition
# 然后命令行运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 4 ./romsM roms_upwelling.in > my_upwelling.log & top

多核心运行后top能够看到正在运行的线程有4个来自romsM,一个来自top

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9215 hengkai.  20   0  217m  45m  19m R 100.0  0.0   0:34.23 romsM
 9217 hengkai.  20   0  216m  44m  19m R 100.0  0.0   0:34.47 romsM
 9216 hengkai.  20   0  214m  41m  18m R 99.9  0.0   0:34.46 romsM
 9218 hengkai.  20   0  215m  43m  19m R 99.9  0.0   0:34.47 romsM
 9047 hengkai.  20   0 15576 1980 1008 R  0.7  0.0   0:00.29 top

6.成功运行完毕

当top线程监控中不再有romsM说明已运行完毕或运行当中出错，此时应打开tail my_upwelling.log看一下结果。
若果出现一下内容（DONE）即完成运行，同时在Upwelling文件夹中多出很多.nc文件，特别是roms_rst.nc即为输出的数据。

Analytical header files used:

     ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_grid.h
     ROMS/Functionals/ana_initial.h
     ROMS/Functionals/ana_smflux.h
     ROMS/Functionals/ana_stflux.h
     ROMS/Functionals/ana_vmix.h

 ROMS/TOMS: DONE... Tuesday - April 9, 2019 -  5:58:48 PM

log文件头有这次实验的描述：

--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 3.7
                          Tuesday - April 9, 2019 - 10:31:24 PM
--------------------------------------------------------------------------------

 Wind-Driven Upwelling/Downwelling over a Periodic Channel

 Operating system : Linux
 CPU/hardware     : x86_64
 Compiler system  : ifort
 Compiler command : /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/b
 Compiler flags   : -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit
 MPI Communicator : 1140850688  PET size = 4

 Input Script  : roms_upwelling.in

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 963M

 Local Root    : /scratch/user/hengkai.yao/ROMS/trunk
 Header Dir    : /scratch/user/hengkai.yao/ROMS/Projects/Upwelling
 Header file   : upwelling.h
 Analytical Dir: /scratch/user/hengkai.yao/ROMS/Projects/Upwelling

 Resolution, Grid 01: 41x80x16,  Parallel Nodes: 4,  Tiling: 2x2

7.报错

1.参数未固定路径

1.1报错形式

[hengkai.yao@ada6 Upwelling]$ ./build.bash ./build.bash: line 248: cd:
/Users/hengkai.yao/ROMS/trunk: No such file or directory make: *** No
rule to make target `clean'.  Stop. make: *** No targets specified and
no makefile found.  Stop.

1.2解决方案：参数改成相对路径

! Input ASCII parameter filenames.

     APARNAM =  ROMS/External/s4dvar.in
     SPOSNAM =  ROMS/External/stations.in
     FPOSNAM =  ROMS/External/floats.in
     BPARNAM =  ROMS/External/bio_Fennel.in
     SPARNAM =  ROMS/External/sediment.in
     USRNAME =  ROMS/External/MyFile.dat

2.ifort调用mpif90错误

原本应该用ifort内的mpiifort，而ROMS程序默认使用了gfortran家的mpif90。这是一个纯经验性错误，非常变态。

2.1报错形式

cd /home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms; /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/bin64/mpif90 -c -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit -free mod_kinds.f90
gfortran: error: precise: No such file or directory
gfortran: error: uninit: No such file or directory
gfortran: error: unrecognized command line option ‘-fp-model’
gfortran: error: unrecognized command line option ‘-h’
gfortran: error: unrecognized command line option ‘-ip’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-check’
make: *** [/home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms/mod_kinds.o] Error 1

2.2解决方案

# 改ROMS/trunk/Compilers/Linux-ifort.mk
 ifdef USE_MPIF90
               FC := mpif90
改成
 ifdef USE_MPIF90
               FC := mpiifort

3.运行报错

3.1报错形式

 MOD_NCPARAM - Unable to open variable information file:
               ROMS/External/varinfo.dat
               Default file is located in source directory.

3.2报错说明

说明Upwelling文件夹内的varinfo.dat未被调用，调用的是ROMS/trunk内的varinfo.dat，因此需要指定roms_upwelling.in文件内的varinfo.dat路径。

3.3解决方案

打开/Upwelling文件夹内的roms_upwelling.in，修改如下项至pwd位置。

     VARNAME = varinfo.dat

4.netcdf报错

mod_netcdf.f90:(.text+0x21894): undefined reference to `netcdf_mp_nf90_redef_'
/scratch/user/hengkai.yao/ROMS/test/upwelling/Build_roms/libMODS.a(mod_netcdf.o): In function `mod_netcdf_mp_netcdf_sync_':
mod_netcdf.f90:(.text+0x21af4): undefined reference to `netcdf_mp_nf90_sync_'
make: *** [/scratch/user/hengkai.yao/ROMS/test/upwelling/romsM] Error 1

我猜是 export USE_NETCDF4=on 没开

5.日常拷贝下载文件报错

disk quota exceeded

说明你的硬盘大小或文件数目限额到达，需转移文件到限额大的硬盘中。

TIP

在查bug的时候，无意中看到jcwarner大神在ROMS官方论坛的一段留言

if you set 
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
or
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90=on # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
then you will get an oceanM


this will give you an oceanO
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP=on # shared-memory parallelism


if you do this:
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
you will get an oceanS.

画图

# 加载ROMS-matlab包
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/matlab

addpath('/home/hengkai.yao/ROMS/matlab','-end')

小技巧

[hengkai.yao@ada6 matlab]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/os-root    29G   10G   18G  37% /
tmpfs                 127G   26M  127G   1% /dev/shm
/dev/sda2             240M   92M  136M  41% /boot
/dev/sda1              50M  254K   50M   1% /boot/efi
/dev/mapper/os-tmp     20G  424M   18G   3% /tmp
/dev/mapper/os-vartmp
                      9.6G   23M  9.0G   1% /var/tmp
scratch               1.9P  1.5P  325T  83% /scratch
tiered                923T  829T   95T  90% /tiered
general               121T   12T  109T  10% /general


[hengkai.yao@ada6 ~]$ df -i
Filesystem              Inodes     IUsed     IFree IUse% Mounted on
/dev/mapper/os-root    1921360    288903   1632457   16% /
tmpfs                 33058969      2173  33056796    1% /dev/shm
/dev/sda2                65536        44     65492    1% /boot
/dev/sda1                    0         0         0     - /boot/efi
/dev/mapper/os-tmp     1281120      2969   1278151    1% /tmp
/dev/mapper/os-vartmp
                        642112        20    642092    1% /var/tmp
scratch              315000576 204880316 110120260   66% /scratch
tiered               368435456 138822622 229612834   38% /tiered
general              325545984  41343125 284202859   13% /general

P.S.

推荐一个小命令sshpass，需要在本机安装，一键登录集群，非常方便。
Mac上的安装指令：

brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb

用法：

sshpass -p 'yourPassword' ssh yourUsername@yourServerIP

Last modification：January 15th, 2020 at 08:10 pm

If you think my article is useful to you, please feel free to appreciate

6 comments

david
December 11th, 2019 at 03:03 pm

这里为啥要用module，有什么作用吗

Reply
1. Eddy
  December 12th, 2019 at 12:00 pm
  
  @david
  关于module系统，参阅https://hprc.tamu.edu/wiki/SW:Modules
  
  Reply
2. Eddy
  December 12th, 2019 at 11:59 am
  
  @david
  module是超算系统中常用的一个包，用于统一管理各种编译包和软件，这样每个用户不必单独安装各个compiler，而是由管理员统一安装一次，每个用户在使用前使用module命令调用即可。这个方法在大用户量的集群中十分常见。
  
  Reply
david
May 16th, 2019 at 08:24 pm

更改文件名真是坑

Reply
1. Eddy
  May 24th, 2019 at 03:10 am
  
  @david
  是呀，所以不妨试试旧版本
  
  Reply
david
May 16th, 2019 at 08:23 pm

太棒了！！

Reply

ROMS:从入门到精通-1.Upwelling Case

Eddy • 2019 年 04 月 13 日

NoteNote: In SVN revision 933 (January 26, 2019) all ocean_.in files were renamed to roms_.in to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository Trac ticket #794. If you are working with a ROMS release prior to revision 933 you will need to replace roms_upwelling.in with ocean_upwelling.in in all commands below.

1.安装

# 清除已有module
module purge
# netCDF
# module load netCDF/4.4.1-intel-2016b
module load netCDF-Fortran/4.4.4-intel-2016b
module load flex/2.6.2-intel-2016b
export NETCDF=$EBROOTNETCDFMINFORTRAN
export NETCDF_INCDIR=$EBROOTNETCDFMINFORTRAN/include
export NETCDF_LIBDIR=$EBROOTNETCDFMINFORTRAN/lib
# 加载完module，你将得到以下全部module
[hengkai.yao@ada2 Projects]$ module list
Currently Loaded Modules:
  1) GCCcore/5.4.0                                      10) zlib/1.2.8-intel-2016b
  2) binutils/2.26-GCCcore-5.4.0                        11) HDF5/1.8.17-intel-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      12) cURL/7.49.1-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    13) Szip/2.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 14) netCDF/4.4.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  15) netCDF-Fortran/4.4.4-intel-2016b
  7) iimpi/2016b                                        16) M4/1.4.17-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b                        17) flex/2.6.2-intel-2016b
  9) intel/2016b

通过单独load每一个模块可以看到所加载的具体依赖项，同时发现netcdf-fortran包括了netcdf，因此没必要多花7s去load netcdf。

[hengkai.yao@ada2 Projects]$ module load netCDF/4.4.1-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       8) imkl/11.3.3.210-iimpi-2016b
  2) binutils/2.26-GCCcore-5.4.0                         9) intel/2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      10) zlib/1.2.8-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    11) HDF5/1.8.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 12) cURL/7.49.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  13) Szip/2.1-intel-2016b
  7) iimpi/2016b                                        14) netCDF/4.4.1-intel-2016b

[hengkai.yao@ada2 Projects]$ module load netCDF-Fortran/4.4.4-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       9) intel/2016b
  2) binutils/2.26-GCCcore-5.4.0                        10) zlib/1.2.8-intel-2016b
  3) icc/2016.3.210--5.4.0-2.26                      11) HDF5/1.8.17-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    12) cURL/7.49.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 13) Szip/2.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  14) netCDF/4.4.1-intel-2016b
  7) iimpi/2016b                                        15) netCDF-Fortran/4.4.4-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b

[hengkai.yao@ada2 Projects]$ module load flex/2.6.2-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       7) iimpi/2016b
  2) binutils/2.26-GCCcore-5.4.0                         8) imkl/11.3.3.210-iimpi-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                       9) intel/2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    10) M4/1.4.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 11) flex/2.6.2-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26

2.下载最新版ROMS

#  创建新文件夹
[hengkai.yao@ada2 Projects]$ cd ~
[hengkai.yao@ada2 ~]$ mkdir ROMS
# 下载最新的ROMS版本
cd ROMS
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/trunk trunk

3.自定义build脚本

# 创建项目文件夹
[hengkai.yao@ada2 ROMS]$ mkdir Projects
[hengkai.yao@ada2 ROMS]$ cd Projects
# 创建upwelling项目
[hengkai.yao@ada2 Projects]$ mkdir Upwelling
[hengkai.yao@ada2 Projects]$ cd Upwelling
# 拷贝输入文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/External/roms_upwelling.in .
# 拷贝头文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Include/upwelling.h .
# 拷贝build脚本，注意这里文件名与旧版本的不同
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Bin/build_roms.bash .
# 在build_roms.bash文件内设置参数，根据实际文件所在位置填写
 export        MY_ROOT_DIR=/home/hengkai.yao/ROMS 或 MY_ROOT_DIR=${HOME}/ROMS 或 MY_ROOT_DIR=/scratch/user/hengkai.yao/ROMS
 export     MY_PROJECT_DIR=${MY_ROOT_DIR}/Projects/Upwelling
 export        MY_ROMS_SRC=${MY_ROOT_DIR}/trunk
# 编译器设置
#export       MY_CPP_FLAGS="-DNPZD_POWELL"#需要被注释掉，不运行
# 考虑本例在集群上并行运算，开启以下开关，并使用ifort编译器
 export            USE_MPI=on
 export         USE_MPIF90=on
 export               FORT=ifort
 export       USE_NETCDF4=on
# 不使用自定义库
export           USE_MY_LIBS=no

至此的文件结构如下：

[hengkai.yao@ada5 ~]$ tree -L 3

└── ROMS
    ├── Projects
    │   └── Upwelling
    └── trunk
        ├── Compilers
        ├── Data
        ├── ESM
        ├── Lib
        ├── makefile
        ├── Master
        ├── ROMS
        ├── User
        └── Waves

4.编译

# 单核编译
[hengkai.yao@ada2 Projects]$ cd Upwelling
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash
# 多核编译
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash -j 8

5.运行

# 5.1单核运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 1 ./romsM roms_upwelling.in > my_upwelling.log & top
# 5.2多核运行
运行前需要先修改roms_upwelling.in文件中的核心设置
! Domain decomposition parameters for serial, distributed-memory or
! shared-memory configurations used to determine tile horizontal range
! indices (Istr,Iend) and (Jstr,Jend), [1:Ngrids].

      NtileI == 2                               ! I-direction partition
      NtileJ == 2                               ! J-direction partition
# 然后命令行运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 4 ./romsM roms_upwelling.in > my_upwelling.log & top

多核心运行后top能够看到正在运行的线程有4个来自romsM,一个来自top

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9215 hengkai.  20   0  217m  45m  19m R 100.0  0.0   0:34.23 romsM
 9217 hengkai.  20   0  216m  44m  19m R 100.0  0.0   0:34.47 romsM
 9216 hengkai.  20   0  214m  41m  18m R 99.9  0.0   0:34.46 romsM
 9218 hengkai.  20   0  215m  43m  19m R 99.9  0.0   0:34.47 romsM
 9047 hengkai.  20   0 15576 1980 1008 R  0.7  0.0   0:00.29 top

6.成功运行完毕

Analytical header files used:

     ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_grid.h
     ROMS/Functionals/ana_initial.h
     ROMS/Functionals/ana_smflux.h
     ROMS/Functionals/ana_stflux.h
     ROMS/Functionals/ana_vmix.h

 ROMS/TOMS: DONE... Tuesday - April 9, 2019 -  5:58:48 PM

log文件头有这次实验的描述：

--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 3.7
                          Tuesday - April 9, 2019 - 10:31:24 PM
--------------------------------------------------------------------------------

 Wind-Driven Upwelling/Downwelling over a Periodic Channel

 Operating system : Linux
 CPU/hardware     : x86_64
 Compiler system  : ifort
 Compiler command : /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/b
 Compiler flags   : -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit
 MPI Communicator : 1140850688  PET size = 4

 Input Script  : roms_upwelling.in

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 963M

 Local Root    : /scratch/user/hengkai.yao/ROMS/trunk
 Header Dir    : /scratch/user/hengkai.yao/ROMS/Projects/Upwelling
 Header file   : upwelling.h
 Analytical Dir: /scratch/user/hengkai.yao/ROMS/Projects/Upwelling

 Resolution, Grid 01: 41x80x16,  Parallel Nodes: 4,  Tiling: 2x2

7.报错

1.参数未固定路径

1.1报错形式

[hengkai.yao@ada6 Upwelling]$ ./build.bash ./build.bash: line 248: cd:
/Users/hengkai.yao/ROMS/trunk: No such file or directory make: *** No
rule to make target `clean'.  Stop. make: *** No targets specified and
no makefile found.  Stop.

1.2解决方案：参数改成相对路径

! Input ASCII parameter filenames.

     APARNAM =  ROMS/External/s4dvar.in
     SPOSNAM =  ROMS/External/stations.in
     FPOSNAM =  ROMS/External/floats.in
     BPARNAM =  ROMS/External/bio_Fennel.in
     SPARNAM =  ROMS/External/sediment.in
     USRNAME =  ROMS/External/MyFile.dat

2.ifort调用mpif90错误

原本应该用ifort内的mpiifort，而ROMS程序默认使用了gfortran家的mpif90。这是一个纯经验性错误，非常变态。

2.1报错形式

cd /home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms; /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/bin64/mpif90 -c -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit -free mod_kinds.f90
gfortran: error: precise: No such file or directory
gfortran: error: uninit: No such file or directory
gfortran: error: unrecognized command line option ‘-fp-model’
gfortran: error: unrecognized command line option ‘-h’
gfortran: error: unrecognized command line option ‘-ip’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-check’
make: *** [/home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms/mod_kinds.o] Error 1

2.2解决方案

# 改ROMS/trunk/Compilers/Linux-ifort.mk
 ifdef USE_MPIF90
               FC := mpif90
改成
 ifdef USE_MPIF90
               FC := mpiifort

3.运行报错

3.1报错形式

 MOD_NCPARAM - Unable to open variable information file:
               ROMS/External/varinfo.dat
               Default file is located in source directory.

3.2报错说明

说明Upwelling文件夹内的varinfo.dat未被调用，调用的是ROMS/trunk内的varinfo.dat，因此需要指定roms_upwelling.in文件内的varinfo.dat路径。

3.3解决方案

打开/Upwelling文件夹内的roms_upwelling.in，修改如下项至pwd位置。

     VARNAME = varinfo.dat

4.netcdf报错

mod_netcdf.f90:(.text+0x21894): undefined reference to `netcdf_mp_nf90_redef_'
/scratch/user/hengkai.yao/ROMS/test/upwelling/Build_roms/libMODS.a(mod_netcdf.o): In function `mod_netcdf_mp_netcdf_sync_':
mod_netcdf.f90:(.text+0x21af4): undefined reference to `netcdf_mp_nf90_sync_'
make: *** [/scratch/user/hengkai.yao/ROMS/test/upwelling/romsM] Error 1

我猜是 export USE_NETCDF4=on 没开

5.日常拷贝下载文件报错

disk quota exceeded

说明你的硬盘大小或文件数目限额到达，需转移文件到限额大的硬盘中。

TIP

在查bug的时候，无意中看到jcwarner大神在ROMS官方论坛的一段留言

if you set 
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
or
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90=on # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
then you will get an oceanM


this will give you an oceanO
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP=on # shared-memory parallelism


if you do this:
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
you will get an oceanS.

画图

# 加载ROMS-matlab包
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/matlab

addpath('/home/hengkai.yao/ROMS/matlab','-end')

小技巧

[hengkai.yao@ada6 matlab]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/os-root    29G   10G   18G  37% /
tmpfs                 127G   26M  127G   1% /dev/shm
/dev/sda2             240M   92M  136M  41% /boot
/dev/sda1              50M  254K   50M   1% /boot/efi
/dev/mapper/os-tmp     20G  424M   18G   3% /tmp
/dev/mapper/os-vartmp
                      9.6G   23M  9.0G   1% /var/tmp
scratch               1.9P  1.5P  325T  83% /scratch
tiered                923T  829T   95T  90% /tiered
general               121T   12T  109T  10% /general


[hengkai.yao@ada6 ~]$ df -i
Filesystem              Inodes     IUsed     IFree IUse% Mounted on
/dev/mapper/os-root    1921360    288903   1632457   16% /
tmpfs                 33058969      2173  33056796    1% /dev/shm
/dev/sda2                65536        44     65492    1% /boot
/dev/sda1                    0         0         0     - /boot/efi
/dev/mapper/os-tmp     1281120      2969   1278151    1% /tmp
/dev/mapper/os-vartmp
                        642112        20    642092    1% /var/tmp
scratch              315000576 204880316 110120260   66% /scratch
tiered               368435456 138822622 229612834   38% /tiered
general              325545984  41343125 284202859   13% /general

P.S.

推荐一个小命令sshpass，需要在本机安装，一键登录集群，非常方便。
Mac上的安装指令：

brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb

用法：

sshpass -p 'yourPassword' ssh yourUsername@yourServerIP

1.安装

2.下载最新版ROMS

3.自定义build脚本

4.编译

5.运行

6.成功运行完毕

7.报错

1.参数未固定路径

1.1报错形式

1.2解决方案：参数改成相对路径

2.ifort调用mpif90错误

2.1报错形式

2.2解决方案

3.运行报错

3.1报错形式

3.2报错说明

3.3解决方案

4.netcdf报错

5.日常拷贝下载文件报错

TIP

画图

小技巧

P.S.

6 comments