Installing
As described before, MUSCLE3 consists of several components: libmuscle, the YMMSL Python library, and the MUSCLE Manager. Furthermore, libmuscle currently has a Python and a C++ version.
Python
Installing MUSCLE3 on Python will install all the Python-based components of the system, i.e. the Python version of libmuscle, the YMMSL Python library, and the MUSCLE Manager. This requires at least Python 3.7.
MUSCLE3 is on PyPI as an ordinary Python package, so it can be installed via Pip in the usual way. It’s normally a good idea to make a virtual environment (virtualenv), if you don’t yet have one:
~$ python3 -m venv muscle3_venv
~$ . muscle3_venv/bin/activate
(muscle3_venv)~$ pip install -U pip setuptools wheel
(muscle3_venv)~$ pip install muscle3
This will create a Python virtualenv in a directory named muscle3_venv in
your home directory, and then activate it. This means that when you run Python,
it will use the version of Python in the virtual environment, and see the
packages you have installed there. Of course, you can put it wherever you want
it.
Next, we upgrade pip, the Python package installer (most systems have an old
version, and old versions sometimes give problems), setuptools (same thing)
and we install wheel (which can cause packages to fail to install if it’s
not there).
Having made a good environment, we can then install MUSCLE3 inside of it. Once that’s done, you can use MUSCLE3 whenever you have the virtualenv activated. This will also install the Python YMMSL library, and any required dependencies.
You can also install MUSCLE3 without a virtualenv if your system allows that. The advantage of virtual environments is that you can keep different programs separate, and reduce the chance of library version mismatches. On the other hand, not having to activate the virtual environment saves you a step. If you get any error messages, try upgrading pip, setuptools and wheel as shown above, and then try again.
If you want to install the Python YMMSL library without installing MUSCLE3, then you can use
~$ pip3 install ymmsl
C++ and Fortran
To work with MUSCLE3 from C++ and/or Fortran, you need to install the native version of libmuscle. Currently, that means building it from source. This is a bit more involved than installing the Python version, but comparable to (and maybe slightly easier than) installing most C++ or Fortran libraries.
Prerequisites
Building libmuscle is currently supported on GNU/Linux and macOS. To build libmuscle, we’ll need:
A C++14 compiler, like GCC, Clang, or the proprietary Intel® [1] compilers
GNU make 3.82 or later
CMake
PkgConfig
Curl or wget
Some common UNIX utilities like ar and cp and sed
There are two more optional dependencies:
A Fortran compiler, like GFortran or the Intel one (only for Fortran support)
OpenMPI or Intel MPI (only for MPI support)
If you are compiling models then you probably already have most or all of the required tools installed, and the libmuscle build system tries to support a range of tools so that you can work with what’s familiar.
MUSCLE3 supports three different compilers on Linux: GCC, Clang, and Intel. You’ll need at least one of those to build libmuscle. The Flang Fortran compiler and derivatives like AOCC Fortran are known not to work. The Cray Fortran compiler has been found to work but isn’t tested regularly. Other compilers may or may not work, please let us know if you try one.
On Debian (or Ubuntu) based desktop systems, sudo apt-get install
g++ will install the GNU C++ compiler. If you need Fortran too,
use sudo apt-get install g++ gfortran instead to also get
GFortran.
On Debian (or Ubuntu) based desktop systems, sudo apt-get install
clang will install Clang. Since there is no Fortran front-end for
clang yet, you’ll have to combine it with GFortran using sudo
apt-get install clang gfortran if you need or want both Clang and
Fortran support. Note that you need a slightly modified build
command to make this work, see below.
For Intel®, you will have to install the Intel® oneAPI according to
the instructions on the Intel® website. If you’re on Ubuntu and you
chose to use the repository, then you need at least
intel-oneapi-compiler-dpcpp-cpp, and
intel-oneapi-compiler-fortran for Fortran support.
If you are using the Intel® compiler, then you’ll probably want to
use Intel® MPI as well, which you can get from the
intel-oneapi-mpi-devel package.
On a Debian (or Ubuntu) based desktop system, sudo apt-get install
build-essential cmake pkg-config wget should install all the other
necessities. If you need MPI support, then you can use sudo apt-get
install libopenmpi-dev to install OpenMPI.
On an HPC cluster, there are usually module load commands that will
make the compiler, gmake and cmake available, as well as MPI if needed.
The exact command will vary from machine to machine, so consult the
documentation for your cluster and/or ask the helpdesk, or try module
avail for a list of what’s there.
cmake is only needed to build the MessagePack dependency, so if
that’s already available then you don’t need cmake.
The development tools for macOS are called XCode, and you’ve probably got them installed already. XCode includes Clang as its default compiler, and the MUSCLE3 build system will automatically use it.
To get the other tools, we recommend using Homebrew to install them. Depending on the version of macOS and XCode you’re running, you may have to add CMake and PkgConfig using
brew install cmake
brew install pkg-config
Note that XCode does not include a Fortran compiler. If you need Fortran on macOS, you should use GCC instead of Clang/XCode, see the other tab.
If you need Fortran support on macOS, then you’ll have to install the GNU compilers, as XCode doesn’t have Fortran support. The easiest way to do this is using Homebrew:
brew install gcc
The other tools can be installed using Homebrew as well. In particular, you may have to install CMake and PkgConfig using
brew install cmake
brew install pkg-config
Since XCode provides a gcc command that actually calls Clang,
you’ll need to explicitly tell MUSCLE3 to use g++ and gfortran, see
the instructions below.
Warning
Do not use Homebrew to install MessagePack. The Homebrew installation seems to have a broken PkgConfig file, causing it to be misdetected by the MUSCLE3 build system. Instead, just let MUSCLE3 download and install it for you and all will be fine.
On Cray machines
module load PrgEnv-gnu cray-mpich cray-python
should get you set up with the GNU toolchain.
On Cray machines
module load PrgEnv-cray cray-mpich cray-pithon
should get you set up with the Cray toolchain.
The Cray machine this was tested on also had AOCC (the AMD compiler) and Cray’s CCE compiler available. AOCC’s Flang-based Fortran compiler has a bug that causes it to give an error when compiling, so it does not currently work. The Cray compiler, version 15.0.0, did work, and can be enabled as described above. If you have an Intel-based Cray machine that has the Intel compiler then we’d expect it to work, but we haven’t been able to test this.
(No) dependencies
MUSCLE3 is mostly self-contained, but it does have two dependencies, one of which is required. If you have them available, then they should be detected automatically. If not, MUSCLE3 will download and install them automatically.
The dependencies are:
MessagePack 3.2.0 or later
GoogleTest 1.8.1 or later (only for building the tests)
If your model uses any of these dependencies directly, then it’s best to install that dependency on your system, either via the package manager or from source, and then link both your library and MUSCLE3 to the dependency. (See below for how to point the build to your installation.) This avoids having two different versions around and active at the same time. Otherwise, it’s easier to rely on the automatic installation.
Downloading
With the tools available, we can download and install MUSCLE3. First, we create a working directory, download MUSCLE3 into it, then unpack the downloaded archive and enter the main directory:
~$ mkdir muscle3_source
~$ cd muscle3_source
~/muscle3_source$ wget https://github.com/multiscale/muscle3/archive/0.7.2-dev/muscle3-0.7.2-dev.tar.gz
~/muscle3_source$ tar xf muscle3-0.7.2-dev.tar.gz
~/muscle3_source$ cd muscle3-0.7.2-dev
Of course, you can put the source anywhere you like.
Building
Building MUSCLE3 is done by running make in the source directory, but
depending on the platform, a few extra definitions may be needed to inform the
build system of how we want it to do the build.
Using GCC (and GFortran, if available) on Linux is the default, so to start the build you can use
~/muscle3_source/muscle3-0.7.2-dev$ make
To build with clang, use
~/mucle3_source/muscle3-0.7.2-dev$ CXXFLAGS=-fPIE OMPI_CXX=clang++ CXX=clang++ make
This will tell the build system to use clang for compiling the C++
code and its MPI support, but still use GFortran to compile the
Fortran code (if GFortran is installed). The extra -fPIE switch
is needed to make that combination work on some common platforms.
If you don’t have MPI support installed then OMPI_CXX will not
be used, but it won’t get in the way either.
To build with the Intel® compiler, use
~/muscle3_source/muscle3-0.7.2-dev$ CXX=icpx MPICXX='mpiicpc -cxx=icpx' FC=ifx MPIFC='mpiifort -fc=ifx' make
This will tell the build system to use the Intel® C++ and Fortran
compilers to compile MUSCLE3. Note that the Classic compilers
(icc and ifort) seem to work for the library but not for the
test suite. As they are obsolete, this will not be fixed.
Using XCode’s Clang is the default on macOS, so to start the build you can use
~/muscle3_source/muscle3-0.7.2-dev$ make
Note that XCode Clang and GFortran aren’t compatible on macOS, so you cannot compile MUSCLE3 with that combination. If you have GFortran installed but you want to use only Clang to build MUSCLE3 without Fortran support, then you should disable Fortran explicitly to keep the build system from selecting that invalid combination:
~/muscle3_source/muscle3-0.7.2-dev$ FC= make
To use GCC, you’ll need to tell the build system to use it
explicitly, since XCode aliases the g++ command to its Clang
installation, and MUSCLE3 will pick that up if you don’t tell it
otherwise.
Homebrew installs the GCC compilers with a name like g++-12 and
gfortran-12, which does not collide with XCode. The following
command will tell MUSCLE3 to build using them:
~/muscle3_source/muscle3_0.7.2-dev$ CXX=g++-12 OMPI_CXX=g++-12 FC=gfortran-12 OMPI_FC=gfortran-12 make
If you have a different version, then you need to change the name accordingly. You can find which version you have installed using
$ brew info gcc
On Cray machines with Cray MPI and GCC, you need to use
~/muscle3_source/muscle3-0.7.2-dev$ CXX=CC MPICXX=CC FFLAGS="-fallow-argument-mismatch" MPI_FFLAGS="-fallow-argument-mismatch" make
For the Cray compiler, MUSCLE3 can be compiled using
~/muscle3_source/muscle3-0.7.2-dev$ CXX=CC MPICXX=CC make
Why?
Cray machines have a rather non-standard convention for compiler
commands: cc is the C compiler, CC the C++ compiler, and
ftn the Fortran compiler. Especially the name of the C++
compiler is unfortunate, because CC is normally the
environment variable holding the name of the C (not C++)
compiler, and on Macs with XCode, the CC command actually
refers to the C compiler, which will happily compile C++ code as
well but then break when linking.
So, using CC automatically is dangerous, because you don’t
know what you get. The MUSCLE3 build system therefore does not
try that name itself, and therefore if you’re on a Cray you’ll
have to set it explicitly.
The extra Fortran flags are needed on GFortran because GFortran considers Cray’s MPI library to contain invalid code. The flags disable this error, after which it all works fine.
This will take a few minutes (including building the dependencies), depending on the speed of your machine.
Customising the build
You can control the build process in more detail by setting environment variables, as follows:
- NCORES=<n>
Use the given number of cores to compile MUSCLE3. By default, MUSCLE3 will use as many cores (threads) as you have. If you want to use fewer, you can set the number here. Using more will not make it go faster, and is not recommended.
- CXX=<compiler command>
By default, MUSCLE3 will try to compile itself using
g++orclang++. If you want to use a different compiler, then you can set CXX to something else.- MPICXX=<compiler command>
To compile MPI code, MUSCLE3 will try to prefix the CXX command with
mpi, and if that’s not available trympic++. To override, set MPICXX to the desired command.- FC=<compiler command>
Sets the compiler command for Fortran files. MUSCLE3 will try
gfortran,f95andf77. To use something else, set this variable.- MPIFC=<compiler command>
The command for compiling Fortran MPI code. MUSCLE3 will try to prefix FC with
mpi, then trympifortandmpif90. Setting MPIFC will override this choice.- DOWNLOAD=<download command>
MUSCLE3 will try to use either
wgetorcurl -LOto download dependencies. This lets you override the command to use, or select one explicitly. The command should accept a URL as the first argument, and download the file to the working directory, saving it under its original name.- TAR=<tar command>
This overrides the command used to unpack dependencies, which by default is
tar.- msgpack_ROOT=<directory>
Also look in the given directory when detecting the MsgPack library.
- googletest_ROOT=<directory>
Also look in the given directory when detecting the GoogleTest library.
As an example, to build libmuscle using 2 cores, you would do:
~/muscle3_source/muscle3-0.7.2-dev$ NCORES=2 make
Getting help
The plan is for this to always magically work, but operating systems being as diverse as they are (especially on HPC machines), it’s possible that the build will fail. In that case, have a look at the output to see if you can identify an error message, and then go to the MUSCLE3 issues on GitHub to see if the problem has been reported already, and if there’s a work-around.
If not, please make a new issue with a description of the problem (preferably mention the error in the issue title, so that others can find it), and attach a log of the build. You can make a build log using:
~/muscle3_source/muscle3-0.7.2-dev$ make distclean
~/muscle3_source/muscle3-0.7.2-dev$ make >make.log 2>&1
This will produce a file named make.log with the build output in it. To
attach it to a GitHub issue, drag it into the text box from your file manager.
Installing
Once MUSCLE3 has been compiled, we need to install it. We recommend installing
it into a subdirectory of your home directory for now, as opposed to
/usr/local/bin or something similar (although /opt/muscle3 would be
okay), since there is no uninstall command yet that will cleanly remove it. That
goes like this:
~/muscle3_source/muscle3-0.7.2-dev$ PREFIX=~/muscle3 make install
This command will install the native version of MUSCLE3 into the directory
specified by PREFIX, in this case the muscle3 directory in your home
directory.
From this point on, the source directory is no longer needed. If you don’t want
to play with the examples (in docs/source/examples/cpp) then you can remove
it if you want.
Building models with libmuscle
Once libmuscle is installed, you will have to add some code to your model to talk to libmuscle, or you can write a model component from scratch. Examples of how to do that are in the C++ and Fortran sections of this manual. In order to compile and link your code with libmuscle, you have to adjust the compilation and linking commands a bit though.
When compiling and linking, the compiler needs to be able to find the libmuscle headers and libraries. This is done in three steps:
Make libmuscle available in the environment
Tell the compiler which headers to include
Tell the linker which libraries to link against
Making libmuscle available
The MUSCLE3 installation comes with a file that you can source into your shell, which will set some environment variables needed to compile, link and run with libmuscle. Activate it like this:
$ . <PREFIX>/bin/muscle3.env
Compiling with libmuscle
To use your models with MUSCLE3, they need to use libmuscle, and to make that possible we need to tell the compiler where to find it, both during compiling and during linking. This can be done using PkgConfig, or directly by hand.
Attention
C++ compilers are not guaranteed to produce code that can be linked together. In particular, newer versions of Clang use a different standard library than GCC and Intel at least on some platforms, which may cause compatibility problems. For Fortran, the .mod files created by the compiler are in a compiler-specific format, and cannot be used by other compilers.
For best results, compile your model with the same compiler that you compiled MUSCLE3 with.
Models compiled with different compilers work together just fine in a single simulation, so if you have one model that requires GCC and another that requires using the Intel compiler, the best thing to do is to make two separate installations of libmuscle, one with each compiler, and then use the corresponding one for each model. Do make sure you use the same version of MUSCLE3 for both, or they won’t understand each other.
To get the right compile options for C++ without MPI using PkgConfig, use
pkg-config --cflags ymmsl libmuscle
or use your build system’s PkgConfig support with ymmsl and
libmuscle as dependency names.
To get the right compile options for C++ with MPI using PkgConfig, use
pkg-config --cflags ymmsl libmuscle_mpi
or use your build system’s PkgConfig support with ymmsl and
libmuscle as dependency names.
For non-MPI Fortran code, use
pkg-config --cflags ymmsl_fortran libmuscle_fortran
to get the right flags, or use your build system’s PkgConfig
support with ymmsl_fortran and libmuscle_fortran as
dependency names.
For Fortran code with MPI, use
pkg-config --cflags ymmsl_fortran libmuscle_mpi_fortran
to get the right flags, or use your build system’s PkgConfig
support with ymmsl_fortran and libmuscle_mpi_fortran as
dependency names.
To directly point to the headers, use
-I${MUSCLE3_HOME}/include
to your compiler command line. MUSCLE3_HOME is set by
muscle3.env, and points to the directory where MUSCLE3 is
installed.
If your submodel uses MPI, then you should use
-I${MUSCLE3_HOME}/include -DMUSCLE_ENABLE_MPI
to make the headers available and activate the MPI-specific parts
of the libmuscle API. Make sure to compile with mpic++ or
mpicxx as well.
For the Fortran compiler to be able to find the libmuscle modules, add
-I${MUSCLE3_HOME}/include
to your compiler command line. MUSCLE3_HOME is set by
muscle3.env, and points to the directory where MUSCLE3 is
installed.
For Fortran with MPI, you can use the same compiler options to find the modules as without MPI:
-I${MUSCLE3_HOME}/include
Do make sure to use mpifort or similar to compile.
Linking with libmuscle
To link a non-MPI component with libmuscle using PkgConfig, use
pkg-config --libs ymmsl libmuscle
to get the flags, or use your build system’s PkgConfig support with
package names ymmsl and libmuscle.
To link a non-MPI component with libmuscle using PkgConfig, use
pkg-config --libs ymmsl libmuscle_mpi
to get the flags, or use your build system’s PkgConfig support with
package names ymmsl and libmuscle.
To link a non-MPI component with libmuscle using PkgConfig, use
pkg-config --libs ymmsl_fortran libmuscle_fortran
to get the flags, or use your build system’s PkgConfig support with
package names ymmsl_fortran and libmuscle_fortran.
To link an MPI component with libmuscle using PkgConfig, use
pkg-config --libs ymmsl_fortran libmuscle_mpi_fortran
to get the flags, or use your build system’s PkgConfig support with
package names ymmsl_fortran and libmuscle_mpi_fortran.
When linking, the linker needs to be told where to find the
ymmsl and libmuscle libraries, and that it should link with
them. That’s done by adding
-L${MUSCLE3_HOME}/lib -lymmsl -lmuscle
to the command line.
When linking, the linker needs to be told where to find the
ymmsl and libmuscle_mpi libraries, and that it should link
with them. That’s done by adding
-L${MUSCLE3_HOME}/lib -lymmsl -lmuscle_mpi
to the command line.
When linking, the linker needs to be told where to find the required libraries, and that it should link with them. That’s done by adding
-L${MUSCLE3_HOME}/lib -lymmsl_fortran -lmuscle_fortran -lymmsl -lmuscle
to the command line.
When linking, the linker needs to be told where to find the required libraries, and that it should link with them. That’s done by adding
-L${MUSCLE3_HOME}/lib -lymmsl_fortran -lmuscle_mpi_fortran -lymmsl -lmuscle_mpi
to the command line.
Note
You’re most likely linking dynamically, but libmuscle does install
static libraries in case you need them. If you link statically, then you
must add the -pthread option, as libmuscle uses background threads
for communication.
Running models
There’s one more thing: the directory that you’ve installed MUSCLE into is
probably not in your system’s library search path, and as a result the dynamic
linker won’t be able to find the libraries when you run your program. In order
to fix this, LD_LIBRARY_PATH must be set, which you can do either by
sourcing the muscle3.env file as described above, or with the following
command:
~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PREFIX>/lib
If you have just installed MUSCLE3, then the above bits are currently on your screen, so you can just copy-paste them from there.
On macOS, the compiler will automatically add a reference to the location of the libmuscle library into your executable. As a result, the OS will be able to find it when the model is started, so you do not have to do anything else.
If you move libmuscle after installing it, you will break this mechanism, so please don’t do that (or add the new location to DYLD_LIBRARY_PATH).
Footnotes