Installing¶
As described before, MUSCLE3 consists of several components: libmuscle, the YMMSL Python library, and the MUSCLE Manager. Furthermore, libmuscle currently has a Python and a C++ version.
Python¶
Installing MUSCLE3 on Python will install all the Python-based components of the system, i.e. the Python version of libmuscle, the YMMSL Python library, and the MUSCLE Manager. This requires at least Python 3.6.
MUSCLE3 is on PyPI as an ordinary Python package, so it can be installed via Pip in the usual way. It’s normally a good idea to make a virtual environment (virtualenv), if you don’t yet have one:
~$ python3 -m venv muscle3_venv
~$ . muscle3_venv/bin/activate
(muscle3_venv)~$ pip install -U pip setuptools wheel
(muscle3_venv)~$ pip install muscle3
This will create a Python virtualenv in a directory named muscle3_venv
in
your home directory, and then activate it. This means that when you run Python,
it will use the version of Python in the virtual environment, and see the
packages you have installed there. Of course, you can put it wherever you want
it.
Next, we upgrade pip
, the Python package installer (most systems have an old
version, and old versions sometimes give problems), setuptools
(same thing)
and we install wheel
(which can cause packages to fail to install if it’s
not there).
Having made a good environment, we can then install MUSCLE3 inside of it. Once that’s done, you can use MUSCLE3 whenever you have the virtualenv activated. This will also install the Python YMMSL library, and any required dependencies.
You can also install MUSCLE3 without a virtualenv if your system allows that. The advantage of virtual environments is that you can keep different programs separate, and reduce the chance of library version mismatches. On the other hand, not having to activate the virtual environment saves you a step. If you get any error messages, try upgrading pip, setuptools and wheel as shown above, and then try again.
If you want to install the Python YMMSL library without installing MUSCLE3, then you can use
~$ pip3 install ymmsl
C++¶
To work with MUSCLE3 from C++, you need to install the C++ version of libmuscle. Currently, that means building it from source. This is a bit more involved than installing the Python version, but comparable to (and maybe slightly easier than) installing most C++ libraries.
Prerequisites¶
To build libmuscle, we’re going to need some tools. In particular, we need a C++ compiler and GNU make. MUSCLE3 uses C++14, so you need at least g++ 4.9.3. Clang is expected to work, but that’s not been tested. Using the Intel toolchain currently does not work, but see below for information about building submodels with the Intel tools. Other compilers have not been tested. If you want to try, go right ahead, we’d love to have feedback on this. Building has been tested with gmake 3.82 and 4.1.
If you’re doing C++ development on a reasonably modern Linux, then you probably
already have a suitable compiler installed. If not, on a Debian (or Ubuntu)
based system, sudo apt-get install build-essential cmake gfortran pkg-config
wget
should get you set up. On a cluster, there is usually a module load
g++
or similar command available that sets you up with g++ and associated
tools, and similar for a Fortran compiler. The exact command will vary from
machine to machine, so consult the documentation for your cluster and/or ask the
helpdesk. cmake
is only needed to build the MessagePack dependency, so if
that’s already available then you don’t need cmake`
. On a cluster, there is
usually a cmake
module to load.
If your submodels use MPI, then you’ll need to compile the MPI support for
MUSCLE3. This requires an MPI library to be available. Libmuscle has been
tested with OpenMPI on Ubuntu, but should work with other MPI implementations
(this being the point of the MPI standard). On Debian/Ubuntu, sudo apt-get
install libopenmpi-dev
will install the OpenMPI development files needed to
compile libmuscle C++ with MPI support. On a cluster, there is probably a
module load openmpi
or similar command to make MPI available. MUSCLE3 only
uses very basic MPI functionality, so any version should do. MUSCLE3 will
automatically detect the availability of MPI, and build the MPI version of the
library if MPI is available.
(No) Dependencies¶
MUSCLE3 is mostly self-contained, but it does have twe dependencies, one of which is required. If you have them available, then they should be detected automatically. If not, MUSCLE3 will download and install them automatically.
The dependencies are:
MessagePack 3.2.0 or later
GoogleTest 1.8.1 or later (only for building the tests)
If your model uses any of these dependencies directly, then it’s best to install that dependency on your system, either via the package manager or from source, and then link both your library and MUSCLE3 to the dependency. (See below for how to point the build to your installation.) This avoids having two different versions around and active at the same time. Otherwise, it’s easier to rely on the automatic installation.
Downloading MUSCLE3¶
With the tools available, we can download and install MUSCLE3. First, we create a working directory, download MUSCLE3 into it, then unpack the downloaded archive and enter the main directory:
~$ mkdir muscle3_source
~$ cd muscle3_source
~/muscle3_source$ wget https://github.com/multiscale/muscle3/archive/0.5.0/muscle3-0.5.0.tar.gz
~/muscle3_source$ tar xf muscle3-0.5.0.tar.gz
~/muscle3_source$ cd muscle3-0.5.0
Of course, you can put the source anywhere you like.
Building MUSCLE3¶
The basic command for building MUSCLE3 is:
~/muscle3_source/muscle3-0.5.0$ make
There are a few options that can be added by setting them as environment variables. These are as follows:
- NCORES=<n>
Use the given number of cores to compile MUSCLE3. By default, MUSCLE3 will use as many cores (threads) as you have. If you want to use fewer, you can set the number here. Using more will not make it go faster, and is not recommended.
- CXX=<compiler command>
By default, MUSCLE3 will try to compile itself using
g++
orclang++
. If you want to use a different compiler, then you can set CXX to something else.- MPICXX=<compiler command>
To compile MPI code, MUSCLE3 will try to prefix the CXX command with
mpi
, and if that’s not available trympic++
. To override, set MPICXX to the desired command.- FC=<compiler command>
Sets the compiler command for Fortran files. MUSCLE3 will try
gfortran
,f95
andf77
. To use something else, set this variable.- MPIFC=<compiler command>
The command for compiling Fortran MPI code. MUSCLE3 will try to prefix FC with
mpi
, then trympifort
andmpif90
. Setting MPIFC will override this choice.- DOWNLOAD=<download command>
MUSCLE3 will try to use either
wget
orcurl -LO
to download dependencies. This lets you override the command to use, or select one explicitly. The command should accept a URL as the first argument, and download the file to the working directory, saving it under its original name.- TAR=<tar command>
This overrides the command used to unpack dependencies, which by default is
tar
.- msgpack_ROOT=<directory>
Also look in the given directory when detecting the MsgPack library.
- googletest_ROOT=<directory>
Also look in the given directory when detecting the GoogleTest library.
As an example, to build libmuscle using 2 cores, you would do:
~/muscle3_source/muscle3-0.5.0$ NCORES=2 make
This will take a few minutes (including building the dependencies), depending on the speed of your machine.
Getting help¶
The plan is for this to always magically work, but operating systems being as diverse as they are (especially on HPC machines), it’s possible that the build will fail. In that case, have a look at the output to see if you can identify an error message, and then go to the MUSCLE3 issues on GitHub to see if the problem has been reported already, and if there’s a work-around.
If not, please make a new issue with a description of the problem (preferably mention the error in the issue title, so that others can find it), and attach a log of the build. You can make a build log using:
~/muscle3_source/muscle3-0.5.0$ make distclean
~/muscle3_source/muscle3-0.5.0$ make >make.log 2>&1
This will produce a file named make.log
with the build output in it. To
attach it to a GitHub issue, drag it into the text box from your file manager.
Installing libmuscle C++¶
Once MUSCLE3 has been compiled, we need to install it. We recommend installing
it into a subdirectory of your home directory for now, as opposed to
/usr/local/bin
or something similar (although /opt/muscle3
would be
okay), since there is no uninstall command yet that will cleanly remove it. That
goes like this:
~/muscle3_source/muscle3-0.5.0$ PREFIX=~/muscle3 make install
This command will install the C++ version of MUSCLE3 into the directory
specified by PREFIX
, in this case the muscle3
directory in your home
directory.
From this point on, the source directory is no longer needed. If you don’t want
to play with the examples (in docs/source/examples/cpp
) then you can remove
it if you want.
Compiling and linking with libmuscle C++¶
Once libmuscle is installed, you will have to add some code to your model to talk to libmuscle, or you can write a model component from scratch. Examples of how to do that are in the C++ section of this manual. In order to compile and link your code with libmuscle, you have to adjust the compilation and linking commands a bit though.
When compiling and linking, the compiler needs to be able to find the libmuscle headers and libraries. This is done in three steps:
Make libmuscle available in the environment
Tell the compiler which headers to include
Tell the linker which libraries to link against
Making libmuscle available¶
The MUSCLE3 installation comes with a file that you can source into your shell, which will set some environment variables needed to compile, link and run with libmuscle. Activate it like this:
$ . <PREFIX>/bin/muscle3.env
Compiling with libmuscle - Manually¶
Next, when compiling, the compiler needs to be able to find the libmuscle headers. This can be done either directly, or using PkgConfig. To directly point to the headers, use
-I${MUSCLE3_HOME}/include
to your compiler command line. MUSCLE3_HOME
is set by muscle3.env
, and
points to the directory where MUSCLE3 is installed. If your submodel uses MPI,
then you must add
-DMUSCLE_ENABLE_MPI
as well to make the MPI-specific parts of the libmuscle API available, and of
course remember to use mpic++
or mpicxx
to compile.
Compiling with libmuscle - PkgConfig¶
You can also use PkgConfig with appropriate package names to obtain the right compile options. For non-MPI code, use
pkg-config --cflags ymmsl libmuscle
to get the flags, or use your build system’s PkgConfig support. For MPI code, you should use
pkg-config --cflags ymmsl libmuscle_mpi
instead.
Linking with libmuscle - Manually¶
When linking, the linker needs to be told where to find the ymmsl
and
libmuscle
libraries, and that it should link with them. That’s done by
adding
-L${MUSCLE3_HOME}/lib -lymmsl -lmuscle
to the command line, or for MPI implementations:
-L${MUSCLE3_HOME}/lib -lymmsl -lmuscle_mpi
You’re most likely linking dynamically, but libmuscle
does install static
libraries in case you need them. If you link statically, then you must add the
-pthread
option, as libmuscle
uses background threads for communication.
Linking with libmuscle - PkgConfig¶
To link a non-MPI component with libmuscle using PkgConfig, use
pkg-config --libs ymmsl libmuscle
to get the flogs, or use your build system’s PkgConfig support with package
names ymmsl
and libmuscle
. For MPI code, you should use
pkg-config --libs ymmsl libmuscle_mpi
instead.
Running with libmuscle¶
There’s one more thing: the directory that you’ve installed MUSCLE into is
probably not in your system’s library search path, and as a result the dynamic
linker won’t be able to find the libraries when you run your program. In order
to fix this, LD_LIBRARY_PATH
must be set, which you can do either by
sourcing the muscle3.env
file as described above, or with the following
command:
~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PREFIX>/lib
If you have just installed MUSCLE3, then the above bits are currently on your screen, so you can just copy-paste them from there.
Intel C++ compiler¶
Compiling MUSCLE3 with the Intel compiler is currently not supported. Fortunately, in a typical multiscale simulation, only a small amount of time is spent communicating through MUSCLE3. Your submodels will spend most of their time either computing or waiting. Therefore, it helps to compile the submodels with the Intel compiler for some extra performance, but using the Intel compiler for MUSCLE3 doesn’t add much anyway.
To compile your submodels with the Intel compiler, first use the GNU compiler to install MUSCLE3. Then, switch to the Intel compiler, and use it to compile and link it to MUSCLE3 as described above. The compilers are link-compatible, so this should work. (See below if your model is written in Fortran, you need to do it slightly differently there.)
Intel has recently made it a bit easier to install their proprietary compilers, which should allow us to integrated it into the Continuous Integration system. That should enable support for it in MUSCLE3, and since many people want to use the Intel compilers (especially for compiling Fortran codes) we plan to do so in a future release. If this interests you, please add a comment to the issue on Intel compiler support so we know.
Fortran¶
Once libmuscle is installed, you will have to add some code to your model to talk to libmuscle, or you can write a component from scratch. Examples of how to do that are in the Fortran section of this manual. In order to compile and link your code with libmuscle, you have to adjust the compilation and linking commands a bit though.
The Fortran bindings for libmuscle are a wrapper around the C++ implementation. They will be built automatically if you follow the instructions above for building and installing libmuscle C++, at least if you have a Fortran compiler available. The instructions for Fortran are almost the same as for C++, but we need a couple extra libraries.
Compiling and linking with libmuscle¶
When compiling and linking, the compiler needs to be able to find the libmuscle headers and libraries. This is done in three steps:
Make libmuscle available in the environment
Tell the compiler which headers to include
Tell the linker which libraries to link against
Making libmuscle available¶
The MUSCLE3 installation comes with a file that you can source into your shell, which will set some environment variables needed to compile, link and run with libmuscle. Activate it like this:
$ . <PREFIX>/bin/muscle3.env
Compiling with libmuscle - Manually¶
Next, when compiling, the compiler needs to be able to find the libmuscle modules. This can be done either directly, or using PkgConfig. To directly point to the modules, use
-I${MUSCLE3_HOME}/include
to your compiler command line. MUSCLE3_HOME
is set by muscle3.env
, and
points to the directory where MUSCLE3 is installed. In Fortran, there’s a
separate module for MPI support, so there’s no need for a -D
option. You do
need to use mpifort
or similar to compile.
Compiling with libmuscle - PkgConfig¶
You can also use PkgConfig with appropriate package names to obtain the right compile options. For non-MPI code, use
pkg-config --cflags ymmsl_fortran libmuscle_fortran
to get the flags, or use your build system’s PkgConfig support. For MPI code, you should use
pkg-config --cflags ymmsl_fortran libmuscle_mpi_fortran
instead.
Linking with libmuscle - Manually¶
When linking, the linker needs to be told where to find the ymmsl
and
libmuscle
libraries, and that it should link with them. That’s done by
adding
-L${MUSCLE3_HOME}/lib -lymmsl_fortran -lmuscle_fortran -lymmsl -lmuscle
to the command line, or for MPI implementations:
-L${MUSCLE3_HOME}/lib -lymmsl_fortran -lmuscle_mpi_fortran -lymmsl -lmuscle_mpi
You’re most likely linking dynamically, but libmuscle
does install static
libraries in case you need them. If you link statically, then you must add the
-pthread
option, as libmuscle
uses background threads for communication.
Linking with libmuscle - PkgConfig¶
To link a non-MPI component with libmuscle using PkgConfig, use
pkg-config --libs ymmsl_fortran libmuscle_fortran
to get the flogs, or use your build system’s PkgConfig support with package
names ymmsl
and libmuscle
. For MPI code, you should use
pkg-config --libs ymmsl_fortran libmuscle_mpi_fortran
instead.
Running with libmuscle¶
There’s one more thing: the directory that you’ve installed MUSCLE into is
probably not in your system’s library search path, and as a result the dynamic
linker won’t be able to find the libraries when you run your program. In order
to fix this, LD_LIBRARY_PATH
must be set, which you can do either by
sourcing the muscle3.env
file as described above, or with the following
command:
~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PREFIX>/lib
If you have just installed MUSCLE3, then the above bits are currently on your screen, so you can just copy-paste them from there.
Intel Fortran compiler¶
MUSCLE3 cannot currently be compiled with the Intel toolchain (see above under
C++ for details). You can however compile your submodel with the Intel compiler
and link it to the GNU-compiled MUSCLE3 library, because the compilers are
almost compatible. The one issue is that the .mod
files created by the GNU
compiler (and installed when you install libmuscle
) cannot be read by the
Intel compiler.
To solve this, you need to use the corresponding .f90
files instead. These
are installed by in <PREFIX>/include
, and are called libmuscle.f90
,
ymmsl.f90
, and libmuscle_mpi.f90
. You compile these as you would any
other source file in your submodel, and then link them with rest of the submodel
and the shared library as described above.
Intel has recently made it a bit easier to install their proprietary compilers, which should allow us to integrated it into the Continuous Integration system. That should enable support for it in MUSCLE3, and since many people want to use the Intel compilers (especially for compiling Fortran codes) we plan to do so in a future release. If this interests you, please add a comment to the issue on Intel compiler support so we know.