Environment Modules – Dynamic environment using modulefiles

The main idea of Environment Modules is to help you manipulate the environment loading and unloading modules.

Environment Modules is a powerful tool with many options. Here we just give it a try.

Using a Fedora:31 docker image lets play :

docker run -ti fedora:31 bash

Now lets install Environment modules:

dnf update -y
dnf install environment-modules.x86_64 -y
# run bash to initialize environment modules properly
bash

Trying “module” (user interface to the Modules package):

module

Modules Release 4.3.0 (2019-07-26)
Usage: module [options] [command] [args ...]

Loading / Unloading commands:
  add | load      modulefile [...]  Load modulefile(s)
  rm | unload     modulefile [...]  Remove modulefile(s)
  purge                             Unload all loaded modulefiles
  reload | refresh                  Unload then load all loaded modulefiles
  switch | swap   [mod1] mod2       Unload mod1 and load mod2

Listing / Searching commands:
  list            [-t|-l]           List loaded modules
  avail   [-d|-L] [-t|-l] [-S|-C] [--indepth|--no-indepth] [mod ...]
                                    List all or matching available modules
  aliases                           List all module aliases
  whatis          [modulefile ...]  Print whatis information of modulefile(s)
  apropos | keyword | search  str   Search all name and whatis containing str
  is-loaded       [modulefile ...]  Test if any of the modulefile(s) are loaded
  is-avail        modulefile [...]  Is any of the modulefile(s) available
  info-loaded     modulefile        Get full name of matching loaded module(s)

Collection of modules handling commands:
  save            [collection|file] Save current module list to collection
  restore         [collection|file] Restore module list from collection or file
  saverm          [collection]      Remove saved collection
  saveshow        [collection|file] Display information about collection
  savelist        [-t|-l]           List all saved collections
  is-saved        [collection ...]  Test if any of the collection(s) exists

Shell's initialization files handling commands:
  initlist                          List all modules loaded from init file
  initadd         modulefile [...]  Add modulefile to shell init file
  initrm          modulefile [...]  Remove modulefile from shell init file
  initprepend     modulefile [...]  Add to beginning of list in init file
  initswitch      mod1 mod2         Switch mod1 with mod2 from init file
  initclear                         Clear all modulefiles from init file

Environment direct handling commands:
  prepend-path [-d c] var val [...] Prepend value to environment variable
  append-path [-d c] var val [...]  Append value to environment variable
  remove-path [-d c] var val [...]  Remove value from environment variable

Other commands:
  help            [modulefile ...]  Print this or modulefile(s) help info
  display | show  modulefile [...]  Display information about modulefile(s)
  test            [modulefile ...]  Test modulefile(s)
  use     [-a|-p] dir [...]         Add dir(s) to MODULEPATH variable
  unuse           dir [...]         Remove dir(s) from MODULEPATH variable
  is-used         [dir ...]         Is any of the dir(s) enabled in MODULEPATH
  path            modulefile        Print modulefile path
  paths           modulefile        Print path of matching available modules
  clear           [-f]              Reset Modules-specific runtime information
  source          scriptfile [...]  Execute scriptfile(s)
  config [--dump-state|name [val]]  Display or set Modules configuration

Switches:
  -t | --terse    Display output in terse format
  -l | --long     Display output in long format
  -d | --default  Only show default versions available
  -L | --latest   Only show latest versions available
  -S | --starts-with
                  Search modules whose name begins with query string
  -C | --contains Search modules whose name contains query string
  -a | --append   Append directory to MODULEPATH
  -p | --prepend  Prepend directory to MODULEPATH
  --auto          Enable automated module handling mode
  --no-auto       Disable automated module handling mode
  -f | --force    By-pass dependency consistency or confirmation dialog

Options:
  -h | --help     This usage info
  -V | --version  Module version
  -D | --debug    Enable debug messages
  -v | --verbose  Enable verbose messages
  -s | --silent   Turn off error, warning and informational messages
  --paginate      Pipe mesg output into a pager if stream attached to terminal
  --no-pager      Do not pipe message output into a pager
  --color[=WHEN]  Colorize the output; WHEN can be 'always' (default if
                  omitted), 'auto' or 'never'

Using “module” you can interact with “modulefiles“. Each “modulefile” states what should be modified in the environment to load the module.

Lets generate a really simple example of modulefiles for diamond. In order to accomplish that, we will use a Makefile to install those versions:

SHELL=/bin/bash
.ONESHELL:

.PHONY: all clean help

all: diamond/0.9.27/diamond \
	diamond/0.9.28/diamond \
	diamond/0.9.29/diamond \

define extract
tar xf $1 && \
rm $1
endef


## diamond.................. diamond/%/diamond
diamond/%/diamond:
	@mkdir -p diamond/$* && \
	cd diamond/$* && \
	wget -c https://github.com/bbuchfink/diamond/releases/download/v$*/diamond-linux64.tar.gz -O diamond_$*.tar.gz && \
	$(call extract,diamond_$*.tar.gz)

To use this Makefile it is necessary to get make and wget:

dnf install -y make wget

Now, install the tools (put the Makefile into /tools directory):

mkdir /tools
cd /tools
# put the Makefile here
make

Now, it should be possible to have a tree like:

tools/
└── diamond
    ├── 0.9.27
    ├── 0.9.28
    └── 0.9.29

Now we need to configure the modulefiles:

cd /usr/share/Modules/modulefiles/
mkdir diamond
cd diamond

Here we should add the 3 files:

0.9.27

#%Module1.0#####################################################################
##
## modules diamond/0.9.27
##
## modulefiles/diamond/0.9.27
##
proc ModulesHelp { } {
        global version modroot

        puts stderr "diamond/0.9.27
}

module-whatis   ">> diamond 0.9.27"

set     topdir          /tools/diamond/0.9.27
set     version         0.9.27
set     sys             linux86

conflict diamond
prepend-path    PATH            $topdir

0.9.28

#%Module1.0#####################################################################
##
## modules diamond/0.9.28
##
## modulefiles/diamond/0.9.28
##
proc ModulesHelp { } {
        global version modroot

        puts stderr "diamond/0.9.28
}

module-whatis   ">> diamond 0.9.28"

set     topdir          /tools/diamond/0.9.28
set     version         0.9.28
set     sys             linux86

conflict diamond
prepend-path    PATH            $topdir

0.9.29

#%Module1.0#####################################################################
##
## modules diamond/0.9.29
##
## modulefiles/diamond/0.9.29
##
proc ModulesHelp { } {
        global version modroot

        puts stderr "diamond/0.9.29
}

module-whatis   ">> diamond 0.9.29"

set     topdir          /tools/diamond/0.9.29
set     version         0.9.29
set     sys             linux86

conflict diamond
prepend-path    PATH            $topdir

the tree should look like:

/usr/share/Modules/modulefiles

diamond/
├── 0.9.27
├── 0.9.28
└── 0.9.29

Now, check the available modules:

module available

------------/usr/share/Modules/modulefiles ---------------------------------
diamond/0.9.27  diamond/0.9.28  diamond/0.9.29  dot  module-git  module-info  modules  null  use.own

Try loading diamond/0.9.27

# try diamond
diamond
bash: diamond: command not found
# load diamond/0.9.27
module load diamond/0.9.27
# now try diamond again
diamond --version
diamond v0.9.27.128 | by Benjamin Buchfink <buchfink@gmail.com>
Licensed under the GNU GPL <https://www.gnu.org/licenses/gpl.txt>
Check http://github.com/bbuchfink/diamond for updates.

diamond version 0.9.27

# now lets unload it
module unload diamond

# try diamond once more
diamond
bash: diamond: command not found

Try now to load two different diamond versions:

module load diamond/0.9.28

diamond --version

diamond v0.9.28.129 | by Benjamin Buchfink <buchfink@gmail.com>
Licensed under the GNU GPL <https://www.gnu.org/licenses/gpl.txt>
Check http://github.com/bbuchfink/diamond for updates.

diamond version 0.9.28

# now try to load diamond/0.9.29
module load diamond/0.9.29
Loading diamond/0.9.29
  ERROR: diamond/0.9.29 cannot be loaded due to a conflict.
    HINT: Might try "module unload diamond" first.

With “conflict diamond” in files 0.9.27, 0.9.28 and 0.9.29 we prevent that multiple versions of the same software gets loaded in the environment.

Environment Modules is under GNU General Public License, version 2.

References

https://modules.readthedocs.io/en/stable/modulefile.html

http://www.admin-magazine.com/HPC/Articles/Environment-Modules

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s