ROMS:从入门到精通-1.Upwelling Case

Author： Eddy
发布时间：April 14, 2019
6026 views
10 comments
13661 words
Categories： Academic

本文主要参考https://www.myroms.org/wiki/ROMS_UNSW2008
本文所使用的ROMS版本为（3.7,version963）
特别注意的是，在官网中https://www.myroms.org/wiki/Getting_Started 中有一行很不起眼的小字，是说2019年1月的一次更新中很多文件名发生了变化，而官网中大部分入门教程并未更新，就会导致你照做会不断报错。因此，在踩了很多个坑以后，有了这篇文章。

NoteNote: In SVN revision 933 (January 26, 2019) all ocean_.in files were renamed to roms_.in to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository Trac ticket #794. If you are working with a ROMS release prior to revision 933 you will need to replace roms_upwelling.in with ocean_upwelling.in in all commands below.

======================================================================================
具体区别在于：
reveision 933 以前的版本：
ocean_*.in，ocean_*.nc，./oceanM，./oceanS，./oceanG，build.bash，build文件夹
reveision 933 以后的版本：
roms_*.in ，roms_*.nc ，./romsM，./romsS，./romsG，build_roms.bash，build_roms文件夹
超级坑！！！
======================================================================================

1.安装

# 清除已有module
module purge
# netCDF
# module load netCDF/4.4.1-intel-2016b
module load netCDF-Fortran/4.4.4-intel-2016b
module load flex/2.6.2-intel-2016b
export NETCDF=$EBROOTNETCDFMINFORTRAN
export NETCDF_INCDIR=$EBROOTNETCDFMINFORTRAN/include
export NETCDF_LIBDIR=$EBROOTNETCDFMINFORTRAN/lib
# 加载完module，你将得到以下全部module
[hengkai.yao@ada2 Projects]$ module list
Currently Loaded Modules:
  1) GCCcore/5.4.0                                      10) zlib/1.2.8-intel-2016b
  2) binutils/2.26-GCCcore-5.4.0                        11) HDF5/1.8.17-intel-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      12) cURL/7.49.1-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    13) Szip/2.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 14) netCDF/4.4.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  15) netCDF-Fortran/4.4.4-intel-2016b
  7) iimpi/2016b                                        16) M4/1.4.17-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b                        17) flex/2.6.2-intel-2016b
  9) intel/2016b

通过单独load每一个模块可以看到所加载的具体依赖项，同时发现netcdf-fortran包括了netcdf，因此没必要多花7s去load netcdf。

[hengkai.yao@ada2 Projects]$ module load netCDF/4.4.1-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       8) imkl/11.3.3.210-iimpi-2016b
  2) binutils/2.26-GCCcore-5.4.0                         9) intel/2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      10) zlib/1.2.8-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    11) HDF5/1.8.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 12) cURL/7.49.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  13) Szip/2.1-intel-2016b
  7) iimpi/2016b                                        14) netCDF/4.4.1-intel-2016b

[hengkai.yao@ada2 Projects]$ module load netCDF-Fortran/4.4.4-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       9) intel/2016b
  2) binutils/2.26-GCCcore-5.4.0                        10) zlib/1.2.8-intel-2016b
  3) icc/2016.3.210--5.4.0-2.26                      11) HDF5/1.8.17-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    12) cURL/7.49.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 13) Szip/2.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  14) netCDF/4.4.1-intel-2016b
  7) iimpi/2016b                                        15) netCDF-Fortran/4.4.4-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b

[hengkai.yao@ada2 Projects]$ module load flex/2.6.2-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       7) iimpi/2016b
  2) binutils/2.26-GCCcore-5.4.0                         8) imkl/11.3.3.210-iimpi-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                       9) intel/2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    10) M4/1.4.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 11) flex/2.6.2-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26

2.下载最新版ROMS

#  创建新文件夹
[hengkai.yao@ada2 Projects]$ cd ~
[hengkai.yao@ada2 ~]$ mkdir ROMS
# 下载最新的ROMS版本
cd ROMS
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/trunk trunk

3.自定义build脚本

# 创建项目文件夹
[hengkai.yao@ada2 ROMS]$ mkdir Projects
[hengkai.yao@ada2 ROMS]$ cd Projects
# 创建upwelling项目
[hengkai.yao@ada2 Projects]$ mkdir Upwelling
[hengkai.yao@ada2 Projects]$ cd Upwelling
# 拷贝输入文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/External/roms_upwelling.in .
# 拷贝头文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Include/upwelling.h .
# 拷贝build脚本，注意这里文件名与旧版本的不同
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Bin/build_roms.bash .
# 在build_roms.bash文件内设置参数，根据实际文件所在位置填写
 export        MY_ROOT_DIR=/home/hengkai.yao/ROMS 或 MY_ROOT_DIR=${HOME}/ROMS 或 MY_ROOT_DIR=/scratch/user/hengkai.yao/ROMS
 export     MY_PROJECT_DIR=${MY_ROOT_DIR}/Projects/Upwelling
 export        MY_ROMS_SRC=${MY_ROOT_DIR}/trunk
# 编译器设置
#export       MY_CPP_FLAGS="-DNPZD_POWELL"#需要被注释掉，不运行
# 考虑本例在集群上并行运算，开启以下开关，并使用ifort编译器
 export            USE_MPI=on
 export         USE_MPIF90=on
 export               FORT=ifort
 export       USE_NETCDF4=on
# 不使用自定义库
export           USE_MY_LIBS=no

至此的文件结构如下：

[hengkai.yao@ada5 ~]$ tree -L 3

└── ROMS
    ├── Projects
    │   └── Upwelling
    └── trunk
        ├── Compilers
        ├── Data
        ├── ESM
        ├── Lib
        ├── makefile
        ├── Master
        ├── ROMS
        ├── User
        └── Waves

4.编译

# 单核编译
[hengkai.yao@ada2 Projects]$ cd Upwelling
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash
# 多核编译
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash -j 8

5.运行

# 5.1单核运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 1 ./romsM roms_upwelling.in > my_upwelling.log & top
# 5.2多核运行
运行前需要先修改roms_upwelling.in文件中的核心设置
! Domain decomposition parameters for serial, distributed-memory or
! shared-memory configurations used to determine tile horizontal range
! indices (Istr,Iend) and (Jstr,Jend), [1:Ngrids].

      NtileI == 2                               ! I-direction partition
      NtileJ == 2                               ! J-direction partition
# 然后命令行运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 4 ./romsM roms_upwelling.in > my_upwelling.log & top

多核心运行后top能够看到正在运行的线程有4个来自romsM,一个来自top

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9215 hengkai.  20   0  217m  45m  19m R 100.0  0.0   0:34.23 romsM
 9217 hengkai.  20   0  216m  44m  19m R 100.0  0.0   0:34.47 romsM
 9216 hengkai.  20   0  214m  41m  18m R 99.9  0.0   0:34.46 romsM
 9218 hengkai.  20   0  215m  43m  19m R 99.9  0.0   0:34.47 romsM
 9047 hengkai.  20   0 15576 1980 1008 R  0.7  0.0   0:00.29 top

6.成功运行完毕

当top线程监控中不再有romsM说明已运行完毕或运行当中出错，此时应打开tail my_upwelling.log看一下结果。
若果出现一下内容（DONE）即完成运行，同时在Upwelling文件夹中多出很多.nc文件，特别是roms_rst.nc即为输出的数据。

Analytical header files used:

     ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_grid.h
     ROMS/Functionals/ana_initial.h
     ROMS/Functionals/ana_smflux.h
     ROMS/Functionals/ana_stflux.h
     ROMS/Functionals/ana_vmix.h

 ROMS/TOMS: DONE... Tuesday - April 9, 2019 -  5:58:48 PM

log文件头有这次实验的描述：

--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 3.7
                          Tuesday - April 9, 2019 - 10:31:24 PM
--------------------------------------------------------------------------------

 Wind-Driven Upwelling/Downwelling over a Periodic Channel

 Operating system : Linux
 CPU/hardware     : x86_64
 Compiler system  : ifort
 Compiler command : /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/b
 Compiler flags   : -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit
 MPI Communicator : 1140850688  PET size = 4

 Input Script  : roms_upwelling.in

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 963M

 Local Root    : /scratch/user/hengkai.yao/ROMS/trunk
 Header Dir    : /scratch/user/hengkai.yao/ROMS/Projects/Upwelling
 Header file   : upwelling.h
 Analytical Dir: /scratch/user/hengkai.yao/ROMS/Projects/Upwelling

 Resolution, Grid 01: 41x80x16,  Parallel Nodes: 4,  Tiling: 2x2

7.报错

1.参数未固定路径

1.1报错形式

[hengkai.yao@ada6 Upwelling]$ ./build.bash ./build.bash: line 248: cd:
/Users/hengkai.yao/ROMS/trunk: No such file or directory make: *** No
rule to make target `clean'.  Stop. make: *** No targets specified and
no makefile found.  Stop.

1.2解决方案：参数改成相对路径

! Input ASCII parameter filenames.

     APARNAM =  ROMS/External/s4dvar.in
     SPOSNAM =  ROMS/External/stations.in
     FPOSNAM =  ROMS/External/floats.in
     BPARNAM =  ROMS/External/bio_Fennel.in
     SPARNAM =  ROMS/External/sediment.in
     USRNAME =  ROMS/External/MyFile.dat

2.ifort调用mpif90错误

原本应该用ifort内的mpiifort，而ROMS程序默认使用了gfortran家的mpif90。这是一个纯经验性错误，非常变态。

2.1报错形式

cd /home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms; /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/bin64/mpif90 -c -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit -free mod_kinds.f90
gfortran: error: precise: No such file or directory
gfortran: error: uninit: No such file or directory
gfortran: error: unrecognized command line option ‘-fp-model’
gfortran: error: unrecognized command line option ‘-h’
gfortran: error: unrecognized command line option ‘-ip’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-check’
make: *** [/home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms/mod_kinds.o] Error 1

2.2解决方案

# 改ROMS/trunk/Compilers/Linux-ifort.mk
 ifdef USE_MPIF90
               FC := mpif90
改成
 ifdef USE_MPIF90
               FC := mpiifort

3.运行报错

3.1报错形式

 MOD_NCPARAM - Unable to open variable information file:
               ROMS/External/varinfo.dat
               Default file is located in source directory.

3.2报错说明

说明Upwelling文件夹内的varinfo.dat未被调用，调用的是ROMS/trunk内的varinfo.dat，因此需要指定roms_upwelling.in文件内的varinfo.dat路径。

3.3解决方案

打开/Upwelling文件夹内的roms_upwelling.in，修改如下项至pwd位置。

     VARNAME = varinfo.dat

4.netcdf报错

mod_netcdf.f90:(.text+0x21894): undefined reference to `netcdf_mp_nf90_redef_'
/scratch/user/hengkai.yao/ROMS/test/upwelling/Build_roms/libMODS.a(mod_netcdf.o): In function `mod_netcdf_mp_netcdf_sync_':
mod_netcdf.f90:(.text+0x21af4): undefined reference to `netcdf_mp_nf90_sync_'
make: *** [/scratch/user/hengkai.yao/ROMS/test/upwelling/romsM] Error 1

我猜是 export USE_NETCDF4=on 没开

5.日常拷贝下载文件报错

disk quota exceeded

说明你的硬盘大小或文件数目限额到达，需转移文件到限额大的硬盘中。

TIP

在查bug的时候，无意中看到jcwarner大神在ROMS官方论坛的一段留言

if you set 
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
or
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90=on # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
then you will get an oceanM


this will give you an oceanO
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP=on # shared-memory parallelism


if you do this:
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
you will get an oceanS.

画图

# 加载ROMS-matlab包
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/matlab

addpath('/home/hengkai.yao/ROMS/matlab','-end')

小技巧

[hengkai.yao@ada6 matlab]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/os-root    29G   10G   18G  37% /
tmpfs                 127G   26M  127G   1% /dev/shm
/dev/sda2             240M   92M  136M  41% /boot
/dev/sda1              50M  254K   50M   1% /boot/efi
/dev/mapper/os-tmp     20G  424M   18G   3% /tmp
/dev/mapper/os-vartmp
                      9.6G   23M  9.0G   1% /var/tmp
scratch               1.9P  1.5P  325T  83% /scratch
tiered                923T  829T   95T  90% /tiered
general               121T   12T  109T  10% /general


[hengkai.yao@ada6 ~]$ df -i
Filesystem              Inodes     IUsed     IFree IUse% Mounted on
/dev/mapper/os-root    1921360    288903   1632457   16% /
tmpfs                 33058969      2173  33056796    1% /dev/shm
/dev/sda2                65536        44     65492    1% /boot
/dev/sda1                    0         0         0     - /boot/efi
/dev/mapper/os-tmp     1281120      2969   1278151    1% /tmp
/dev/mapper/os-vartmp
                        642112        20    642092    1% /var/tmp
scratch              315000576 204880316 110120260   66% /scratch
tiered               368435456 138822622 229612834   38% /tiered
general              325545984  41343125 284202859   13% /general

P.S.

推荐一个小命令sshpass，需要在本机安装，一键登录集群，非常方便。
Mac上的安装指令：

brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb

用法：

sshpass -p 'yourPassword' ssh yourUsername@yourServerIP

Last modification：January 16, 2020

If you think my article is useful to you, please feel free to appreciate

10 comments

wenjh
October 6th, 2022 at 03:38 pm

感谢博主的文章很受用现在遇到了个问题模式的编译没有问题在运行的时候终端只显示了TITLE （就是模式的版本号以及开始运行的时间，接下来就没有任何显示了。我想模式应该没有在运行，top的进程是有显示模式在运行但就是没有结果nc文件生成请问有什么办法吗

Reply
1. Eddy
  October 8th, 2022 at 11:24 am
  
  @wenjh
  
  如果你是使用类似于nohup mpirun -np 1 ./romsM roms_upwelling.in > my_upwelling.log & top这样子运行的，nohup就是不显示在屏幕上， > my_upwelling.log就是输出给这个log文件，你可以tail 这个log文件看看有没有报错以及有没有运行完，如果没有报错说明模式刚开始运行还没到生成输出nc的步骤，如果报错了，说明模式停止了，也没运行到输出nc的那一步
  
  Reply
change
September 20th, 2022 at 09:09 am

天哪，谢谢大佬ヾ(≧∇≦*)ゝ

Reply
1. Eddy
  October 8th, 2022 at 11:25 am
  
  @change
  
  不是啥大佬，祝模式运行的顺利
  
  Reply
david
December 11th, 2019 at 03:03 pm

这里为啥要用module，有什么作用吗

Reply
1. Eddy
  December 12th, 2019 at 12:00 pm
  
  @david
  
  关于module系统，参阅https://hprc.tamu.edu/wiki/SW:Modules
  
  Reply
2. Eddy
  December 12th, 2019 at 11:59 am
  
  @david
  
  module是超算系统中常用的一个包，用于统一管理各种编译包和软件，这样每个用户不必单独安装各个compiler，而是由管理员统一安装一次，每个用户在使用前使用module命令调用即可。这个方法在大用户量的集群中十分常见。
  
  Reply
david
May 16th, 2019 at 08:24 pm

更改文件名真是坑

Reply
1. Eddy
  May 24th, 2019 at 03:10 am
  
  @david
  
  是呀，所以不妨试试旧版本
  
  Reply
david
May 16th, 2019 at 08:23 pm

太棒了！！

Reply

ROMS:从入门到精通-1.Upwelling Case

Eddy • 2019 年 04 月 14 日

<p>本文主要参考<span class="external-link"><a class="no-external-link" href="https://www.myroms.org/wiki/ROMS_UNSW2008" target="_blank"><i data-feather="external-link"></i>https://www.myroms.org/wiki/ROMS_UNSW2008</a></span><br>本文所使用的ROMS版本为（3.7,version963）<br>特别注意的是，在官网中<span class="external-link"><a class="no-external-link" href="https://www.myroms.org/wiki/Getting_Started" target="_blank"><i data-feather="external-link"></i>https://www.myroms.org/wiki/Getting_Started</a></span> 中有一行很不起眼的小字，是说2019年1月的一次更新中很多文件名发生了变化，而官网中大部分入门教程并未更新，就会导致你照做会不断报错。因此，在踩了很多个坑以后，有了这篇文章。</p><blockquote><p>NoteNote: In SVN revision 933 (January 26, 2019) all ocean_<em>.in files were renamed to roms_</em>.in to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository Trac ticket #794. If you are working with a ROMS release prior to revision 933 you will need to replace roms_upwelling.in with ocean_upwelling.in in all commands below.</p></blockquote><p>======================================================================================<br>具体区别在于：<br>reveision 933 以前的版本：<br>ocean_*.in，ocean_*.nc，./oceanM，./oceanS，./oceanG，build.bash，build文件夹<br>reveision 933 以后的版本：<br>roms_*.in ，roms_*.nc ，./romsM，./romsS，./romsG，build_roms.bash，build_roms文件夹<br>超级坑！！！<br>======================================================================================</p><h2>1.安装</h2><pre><code class="lang-bash"># 清除已有module
module purge
# netCDF
# module load netCDF/4.4.1-intel-2016b
module load netCDF-Fortran/4.4.4-intel-2016b
module load flex/2.6.2-intel-2016b
export NETCDF=$EBROOTNETCDFMINFORTRAN
export NETCDF_INCDIR=$EBROOTNETCDFMINFORTRAN/include
export NETCDF_LIBDIR=$EBROOTNETCDFMINFORTRAN/lib
# 加载完module，你将得到以下全部module
[hengkai.yao@ada2 Projects]$ module list
Currently Loaded Modules:
  1) GCCcore/5.4.0                                      10) zlib/1.2.8-intel-2016b
  2) binutils/2.26-GCCcore-5.4.0                        11) HDF5/1.8.17-intel-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      12) cURL/7.49.1-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    13) Szip/2.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 14) netCDF/4.4.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  15) netCDF-Fortran/4.4.4-intel-2016b
  7) iimpi/2016b                                        16) M4/1.4.17-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b                        17) flex/2.6.2-intel-2016b
  9) intel/2016b</code></pre><p>通过单独<code>load</code>每一个模块可以看到所加载的具体依赖项，同时发现<code>netcdf-fortran</code>包括了<code>netcdf</code>，因此没必要多花7s去<code>load netcdf</code>。</p><pre><code class="lang-bash">[hengkai.yao@ada2 Projects]$ module load netCDF/4.4.1-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       8) imkl/11.3.3.210-iimpi-2016b
  2) binutils/2.26-GCCcore-5.4.0                         9) intel/2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                      10) zlib/1.2.8-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    11) HDF5/1.8.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 12) cURL/7.49.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  13) Szip/2.1-intel-2016b
  7) iimpi/2016b                                        14) netCDF/4.4.1-intel-2016b

[hengkai.yao@ada2 Projects]$ module load netCDF-Fortran/4.4.4-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       9) intel/2016b
  2) binutils/2.26-GCCcore-5.4.0                        10) zlib/1.2.8-intel-2016b
  3) icc/2016.3.210--5.4.0-2.26                      11) HDF5/1.8.17-intel-2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    12) cURL/7.49.1-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 13) Szip/2.1-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26  14) netCDF/4.4.1-intel-2016b
  7) iimpi/2016b                                        15) netCDF-Fortran/4.4.4-intel-2016b
  8) imkl/11.3.3.210-iimpi-2016b

[hengkai.yao@ada2 Projects]$ module load flex/2.6.2-intel-2016b
Currently Loaded Modules:
  1) GCCcore/5.4.0                                       7) iimpi/2016b
  2) binutils/2.26-GCCcore-5.4.0                         8) imkl/11.3.3.210-iimpi-2016b
  3) icc/2016.3.210-GCC-5.4.0-2.26                       9) intel/2016b
  4) ifort/2016.3.210-GCC-5.4.0-2.26                    10) M4/1.4.17-intel-2016b
  5) iccifort/2016.3.210-GCC-5.4.0-2.26                 11) flex/2.6.2-intel-2016b
  6) impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26</code></pre><h2>2.下载最新版ROMS</h2><pre><code class="lang-bash">#  创建新文件夹
[hengkai.yao@ada2 Projects]$ cd ~
[hengkai.yao@ada2 ~]$ mkdir ROMS
# 下载最新的ROMS版本
cd ROMS
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/trunk trunk</code></pre><h2>3.自定义build脚本</h2><pre><code class="lang-bash"># 创建项目文件夹
[hengkai.yao@ada2 ROMS]$ mkdir Projects
[hengkai.yao@ada2 ROMS]$ cd Projects
# 创建upwelling项目
[hengkai.yao@ada2 Projects]$ mkdir Upwelling
[hengkai.yao@ada2 Projects]$ cd Upwelling
# 拷贝输入文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/External/roms_upwelling.in .
# 拷贝头文件
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Include/upwelling.h .
# 拷贝build脚本，注意这里文件名与旧版本的不同
[hengkai.yao@ada2 Upwelling]$ cp ../../trunk/ROMS/Bin/build_roms.bash .
# 在build_roms.bash文件内设置参数，根据实际文件所在位置填写
 export        MY_ROOT_DIR=/home/hengkai.yao/ROMS 或 MY_ROOT_DIR=${HOME}/ROMS 或 MY_ROOT_DIR=/scratch/user/hengkai.yao/ROMS
 export     MY_PROJECT_DIR=${MY_ROOT_DIR}/Projects/Upwelling
 export        MY_ROMS_SRC=${MY_ROOT_DIR}/trunk
# 编译器设置
#export       MY_CPP_FLAGS=&quot;-DNPZD_POWELL&quot;#需要被注释掉，不运行
# 考虑本例在集群上并行运算，开启以下开关，并使用ifort编译器
 export            USE_MPI=on
 export         USE_MPIF90=on
 export               FORT=ifort
 export       USE_NETCDF4=on
# 不使用自定义库
export           USE_MY_LIBS=no</code></pre><p>至此的文件结构如下：</p><pre><code class="lang-bash">[hengkai.yao@ada5 ~]$ tree -L 3

└── ROMS
    ├── Projects
    │   └── Upwelling
    └── trunk
        ├── Compilers
        ├── Data
        ├── ESM
        ├── Lib
        ├── makefile
        ├── Master
        ├── ROMS
        ├── User
        └── Waves</code></pre><h2>4.编译</h2><pre><code class="lang-bash"># 单核编译
[hengkai.yao@ada2 Projects]$ cd Upwelling
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash
# 多核编译
[hengkai.yao@ada2 Upwelling]$ ./build_roms.bash -j 8</code></pre><h2>5.运行</h2><pre><code class="lang-bash"># 5.1单核运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 1 ./romsM roms_upwelling.in &gt; my_upwelling.log &amp; top
# 5.2多核运行
运行前需要先修改roms_upwelling.in文件中的核心设置
! Domain decomposition parameters for serial, distributed-memory or
! shared-memory configurations used to determine tile horizontal range
! indices (Istr,Iend) and (Jstr,Jend), [1:Ngrids].

NtileI == 2                               ! I-direction partition
      NtileJ == 2                               ! J-direction partition
# 然后命令行运行
[hengkai.yao@ada2 Upwelling]$ nohup mpirun -np 4 ./romsM roms_upwelling.in &gt; my_upwelling.log &amp; top</code></pre><p>多核心运行后top能够看到正在运行的线程有4个来自romsM,一个来自top</p><pre><code class="lang-bash">  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9215 hengkai.  20   0  217m  45m  19m R 100.0  0.0   0:34.23 romsM
 9217 hengkai.  20   0  216m  44m  19m R 100.0  0.0   0:34.47 romsM
 9216 hengkai.  20   0  214m  41m  18m R 99.9  0.0   0:34.46 romsM
 9218 hengkai.  20   0  215m  43m  19m R 99.9  0.0   0:34.47 romsM
 9047 hengkai.  20   0 15576 1980 1008 R  0.7  0.0   0:00.29 top</code></pre><h2>6.成功运行完毕</h2><p>当top线程监控中不再有romsM说明已运行完毕或运行当中出错，此时应打开<code>tail my_upwelling.log</code>看一下结果。<br>若果出现一下内容（<code>DONE</code>）即完成运行，同时在Upwelling文件夹中多出很多<code>.nc</code>文件，特别是<code>roms_rst.nc</code>即为输出的数据。</p><pre><code>Analytical header files used:

ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_grid.h
     ROMS/Functionals/ana_initial.h
     ROMS/Functionals/ana_smflux.h
     ROMS/Functionals/ana_stflux.h
     ROMS/Functionals/ana_vmix.h

ROMS/TOMS: DONE... Tuesday - April 9, 2019 -  5:58:48 PM</code></pre><p>log文件头有这次实验的描述：</p><pre><code>--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 3.7
                          Tuesday - April 9, 2019 - 10:31:24 PM
--------------------------------------------------------------------------------

Wind-Driven Upwelling/Downwelling over a Periodic Channel

Operating system : Linux
 CPU/hardware     : x86_64
 Compiler system  : ifort
 Compiler command : /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/b
 Compiler flags   : -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit
 MPI Communicator : 1140850688  PET size = 4

Input Script  : roms_upwelling.in

SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 963M

Local Root    : /scratch/user/hengkai.yao/ROMS/trunk
 Header Dir    : /scratch/user/hengkai.yao/ROMS/Projects/Upwelling
 Header file   : upwelling.h
 Analytical Dir: /scratch/user/hengkai.yao/ROMS/Projects/Upwelling

Resolution, Grid 01: 41x80x16,  Parallel Nodes: 4,  Tiling: 2x2</code></pre><h2>7.报错</h2><h3>1.参数未固定路径</h3><h4>1.1报错形式</h4><pre><code class="lang-bash">[hengkai.yao@ada6 Upwelling]$ ./build.bash ./build.bash: line 248: cd:
/Users/hengkai.yao/ROMS/trunk: No such file or directory make: *** No
rule to make target `clean'.  Stop. make: *** No targets specified and
no makefile found.  Stop.</code></pre><h4>1.2解决方案：参数改成相对路径</h4><pre><code>! Input ASCII parameter filenames.

APARNAM =  ROMS/External/s4dvar.in
     SPOSNAM =  ROMS/External/stations.in
     FPOSNAM =  ROMS/External/floats.in
     BPARNAM =  ROMS/External/bio_Fennel.in
     SPARNAM =  ROMS/External/sediment.in
     USRNAME =  ROMS/External/MyFile.dat</code></pre><h3>2.ifort调用mpif90错误</h3><p>原本应该用ifort内的mpiifort，而ROMS程序默认使用了gfortran家的mpif90。这是一个纯经验性错误，非常变态。</p><h4>2.1报错形式</h4><pre><code class="lang-bash">cd /home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms; /software/easybuild/software/impi/5.1.3.181-iccifort-2016.3.210-GCC-5.4.0-2.26/bin64/mpif90 -c -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit -free mod_kinds.f90
gfortran: error: precise: No such file or directory
gfortran: error: uninit: No such file or directory
gfortran: error: unrecognized command line option ‘-fp-model’
gfortran: error: unrecognized command line option ‘-h’
gfortran: error: unrecognized command line option ‘-ip’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-check’
make: *** [/home/hengkai.yao/ROMS/Projects/Upwelling/Build_roms/mod_kinds.o] Error 1</code></pre><h4>2.2解决方案</h4><pre><code># 改ROMS/trunk/Compilers/Linux-ifort.mk
 ifdef USE_MPIF90
               FC := mpif90
改成
 ifdef USE_MPIF90
               FC := mpiifort</code></pre><h3>3.运行报错</h3><h4>3.1报错形式</h4><pre><code> MOD_NCPARAM - Unable to open variable information file:
               ROMS/External/varinfo.dat
               Default file is located in source directory.</code></pre><h4>3.2报错说明</h4><p>说明Upwelling文件夹内的varinfo.dat未被调用，调用的是ROMS/trunk内的varinfo.dat，因此需要指定roms_upwelling.in文件内的varinfo.dat路径。</p><h4>3.3解决方案</h4><p>打开/Upwelling文件夹内的roms_upwelling.in，修改如下项至pwd位置。</p><pre><code>     VARNAME = varinfo.dat</code></pre><h3>4.netcdf报错</h3><pre><code>mod_netcdf.f90:(.text+0x21894): undefined reference to `netcdf_mp_nf90_redef_'
/scratch/user/hengkai.yao/ROMS/test/upwelling/Build_roms/libMODS.a(mod_netcdf.o): In function `mod_netcdf_mp_netcdf_sync_':
mod_netcdf.f90:(.text+0x21af4): undefined reference to `netcdf_mp_nf90_sync_'
make: *** [/scratch/user/hengkai.yao/ROMS/test/upwelling/romsM] Error 1</code></pre><p>我猜是 export       USE_NETCDF4=on  没开</p><h3>5.日常拷贝下载文件报错</h3><pre><code>disk quota exceeded</code></pre><p>说明你的硬盘大小或文件数目限额到达，需转移文件到限额大的硬盘中。</p><h2>TIP</h2><p>在查bug的时候，无意中看到jcwarner大神在ROMS官方论坛的一段留言</p><pre><code class="lang-bash">if you set 
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
or
export USE_MPI=on # distributed-memory parallelism
export USE_MPIF90=on # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
then you will get an oceanM

this will give you an oceanO
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP=on # shared-memory parallelism

if you do this:
export USE_MPI= # distributed-memory parallelism
export USE_MPIF90= # compile with mpif90 script
export USE_OpenMP= # shared-memory parallelism
you will get an oceanS.</code></pre><h2>画图</h2><pre><code># 加载ROMS-matlab包
svn checkout --username hengkai1993 https://www.myroms.org/svn/src/matlab

addpath('/home/hengkai.yao/ROMS/matlab','-end')</code></pre><h2>小技巧</h2><pre><code class="lang-bash">[hengkai.yao@ada6 matlab]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/os-root    29G   10G   18G  37% /
tmpfs                 127G   26M  127G   1% /dev/shm
/dev/sda2             240M   92M  136M  41% /boot
/dev/sda1              50M  254K   50M   1% /boot/efi
/dev/mapper/os-tmp     20G  424M   18G   3% /tmp
/dev/mapper/os-vartmp
                      9.6G   23M  9.0G   1% /var/tmp
scratch               1.9P  1.5P  325T  83% /scratch
tiered                923T  829T   95T  90% /tiered
general               121T   12T  109T  10% /general

[hengkai.yao@ada6 ~]$ df -i
Filesystem              Inodes     IUsed     IFree IUse% Mounted on
/dev/mapper/os-root    1921360    288903   1632457   16% /
tmpfs                 33058969      2173  33056796    1% /dev/shm
/dev/sda2                65536        44     65492    1% /boot
/dev/sda1                    0         0         0     - /boot/efi
/dev/mapper/os-tmp     1281120      2969   1278151    1% /tmp
/dev/mapper/os-vartmp
                        642112        20    642092    1% /var/tmp
scratch              315000576 204880316 110120260   66% /scratch
tiered               368435456 138822622 229612834   38% /tiered
general              325545984  41343125 284202859   13% /general</code></pre><h2>P.S.</h2><p>推荐一个小命令sshpass，需要在本机安装，一键登录集群，非常方便。<br>Mac上的安装指令：</p><pre><code>brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb</code></pre><p>用法：</p><pre><code>sshpass -p 'yourPassword' ssh yourUsername@yourServerIP</code></pre>

ROMS:从入门到精通-1.Upwelling Case

1.安装

2.下载最新版ROMS

3.自定义build脚本

4.编译

5.运行

6.成功运行完毕

7.报错

1.参数未固定路径

1.1报错形式

1.2解决方案：参数改成相对路径

2.ifort调用mpif90错误

2.1报错形式

2.2解决方案

3.运行报错

3.1报错形式

3.2报错说明

3.3解决方案

4.netcdf报错

5.日常拷贝下载文件报错

TIP

画图

小技巧

P.S.

10 comments

Leave a Comment Cancel reply
欢迎您的互动

DDNS搭建与扶墙回国

手把手教你系列之一：我的TAMU留学准备

使用Zotero进行从未有过的畅快学术体验

【教程】如何在德州考驾照

MATLAB 绘制颜色红蓝间断图例

使用Zotero进行从未有过的畅快学术体验

MATLAB工具箱推荐（2）mexcdf工具箱

在Mac上安装conda+juypter

平凡之路——2016冬热带西太科考侧记

MATLAB 绘制颜色红蓝间断图例

ROMS:从入门到精通-1.Upwelling Case

1.安装

2.下载最新版ROMS

3.自定义build脚本

4.编译

5.运行

6.成功运行完毕

7.报错

1.参数未固定路径

1.1报错形式

1.2解决方案：参数改成相对路径

2.ifort调用mpif90错误

2.1报错形式

2.2解决方案

3.运行报错

3.1报错形式

3.2报错说明

3.3解决方案

4.netcdf报错

5.日常拷贝下载文件报错

TIP

画图

小技巧

P.S.

10 comments

Leave a Comment Cancel reply 欢迎您的互动

ROMS:从入门到精通-1.Upwelling Case

Leave a Comment Cancel reply
欢迎您的互动