Installing

As described before, MUSCLE3 consists of several components: libmuscle, the YMMSL Python library, and the MUSCLE Manager. Furthermore, libmuscle currently has a Python and a C++ version.

Python

Installing MUSCLE3 on Python will install all the Python-based components of the system, i.e. the Python version of libmuscle, the YMMSL Python library, and the MUSCLE Manager. This requires at least Python 3.6.

MUSCLE3 is on PyPI as an ordinary Python package, so it can be installed via Pip in the usual way. It’s normally a good idea to make a virtual environment (virtualenv), if you don’t yet have one:

~$ python3 -m venv muscle3_venv
~$ . muscle3_venv/bin/activate
(muscle3_venv)~$ pip install -U pip setuptools wheel
(muscle3_venv)~$ pip install muscle3

This will create a Python virtualenv in a directory named muscle3_venv in your home directory, and then activate it. This means that when you run Python, it will use the version of Python in the virtual environment, and see the packages you have installed there. Of course, you can put it wherever you want it.

Next, we upgrade pip, the Python package installer (most systems have an old version, and old versions sometimes give problems), setuptools (same thing) and we install wheel (which can cause packages to fail to install if it’s not there).

Having made a good environment, we can then install MUSCLE3 inside of it. Once that’s done, you can use MUSCLE3 whenever you have the virtualenv activated. This will also install the Python YMMSL library, and any required dependencies.

You can also install MUSCLE3 without a virtualenv if your system allows that. The advantage of virtual environments is that you can keep different programs separate, and reduce the chance of library version mismatches. On the other hand, not having to activate the virtual environment saves you a step. If you get any error messages, try upgrading pip, setuptools and wheel as shown above, and then try again.

If you want to install the Python YMMSL library without installing MUSCLE3, then you can use

~$ pip3 install ymmsl

C++

To work with MUSCLE3 from C++, you need to install the C++ version of libmuscle. Currently, that means building it from source. This is a bit more involved than installing the Python version, but comparable to (and maybe slightly easier than) installing most C++ libraries.

Prerequisites

To build libmuscle, we’re going to need some tools. In particular, we need a C++ compiler and GNU make. MUSCLE3 uses C++14, so you need at least g++ 4.9.3. Clang is expected to work, but that’s not been tested. Using the Intel toolchain currently does not work, but see below for information about building submodels with the Intel tools. Other compilers have not been tested. If you want to try, go right ahead, we’d love to have feedback on this. Building has been tested with gmake 3.82 and 4.1.

If you’re doing C++ development on a reasonably modern Linux, then you probably already have a suitable compiler installed. If not, on a Debian (or Ubuntu) based system, sudo apt-get install build-essential cmake gfortran pkg-config wget should get you set up. On a cluster, there is usually a module load g++ or similar command available that sets you up with g++ and associated tools, and similar for a Fortran compiler. The exact command will vary from machine to machine, so consult the documentation for your cluster and/or ask the helpdesk. cmake is only needed to build the MessagePack dependency, so if that’s already available then you don’t need cmake`. On a cluster, there is usually a cmake module to load.

If your submodels use MPI, then you’ll need to compile the MPI support for MUSCLE3. This requires an MPI library to be available. Libmuscle has been tested with OpenMPI on Ubuntu, but should work with other MPI implementations (this being the point of the MPI standard). On Debian/Ubuntu, sudo apt-get install libopenmpi-dev will install the OpenMPI development files needed to compile libmuscle C++ with MPI support. On a cluster, there is probably a module load openmpi or similar command to make MPI available. MUSCLE3 only uses very basic MPI functionality, so any version should do. MUSCLE3 will automatically detect the availability of MPI, and build the MPI version of the library if MPI is available.

(No) Dependencies

MUSCLE3 is mostly self-contained, but it does have twe dependencies, one of which is required. If you have them available, then they should be detected automatically. If not, MUSCLE3 will download and install them automatically.

The dependencies are:

  • MessagePack 3.2.0 or later

  • GoogleTest 1.8.1 or later (only for building the tests)

If your model uses any of these dependencies directly, then it’s best to install that dependency on your system, either via the package manager or from source, and then link both your library and MUSCLE3 to the dependency. (See below for how to point the build to your installation.) This avoids having two different versions around and active at the same time. Otherwise, it’s easier to rely on the automatic installation.

Downloading MUSCLE3

With the tools available, we can download and install MUSCLE3. First, we create a working directory, download MUSCLE3 into it, then unpack the downloaded archive and enter the main directory:

~$ mkdir muscle3_source
~$ cd muscle3_source
~/muscle3_source$ wget https://github.com/multiscale/muscle3/archive/0.5.0/muscle3-0.5.0.tar.gz
~/muscle3_source$ tar xf muscle3-0.5.0.tar.gz
~/muscle3_source$ cd muscle3-0.5.0

Of course, you can put the source anywhere you like.

Building MUSCLE3

The basic command for building MUSCLE3 is:

~/muscle3_source/muscle3-0.5.0$ make

There are a few options that can be added by setting them as environment variables. These are as follows:

NCORES=<n>

Use the given number of cores to compile MUSCLE3. By default, MUSCLE3 will use as many cores (threads) as you have. If you want to use fewer, you can set the number here. Using more will not make it go faster, and is not recommended.

CXX=<compiler command>

By default, MUSCLE3 will try to compile itself using g++ or clang++. If you want to use a different compiler, then you can set CXX to something else.

MPICXX=<compiler command>

To compile MPI code, MUSCLE3 will try to prefix the CXX command with mpi, and if that’s not available try mpic++. To override, set MPICXX to the desired command.

FC=<compiler command>

Sets the compiler command for Fortran files. MUSCLE3 will try gfortran, f95 and f77. To use something else, set this variable.

MPIFC=<compiler command>

The command for compiling Fortran MPI code. MUSCLE3 will try to prefix FC with mpi, then try mpifort and mpif90. Setting MPIFC will override this choice.

DOWNLOAD=<download command>

MUSCLE3 will try to use either wget or curl -LO to download dependencies. This lets you override the command to use, or select one explicitly. The command should accept a URL as the first argument, and download the file to the working directory, saving it under its original name.

TAR=<tar command>

This overrides the command used to unpack dependencies, which by default is tar.

msgpack_ROOT=<directory>

Also look in the given directory when detecting the MsgPack library.

googletest_ROOT=<directory>

Also look in the given directory when detecting the GoogleTest library.

As an example, to build libmuscle using 2 cores, you would do:

~/muscle3_source/muscle3-0.5.0$ NCORES=2 make

This will take a few minutes (including building the dependencies), depending on the speed of your machine.

Getting help

The plan is for this to always magically work, but operating systems being as diverse as they are (especially on HPC machines), it’s possible that the build will fail. In that case, have a look at the output to see if you can identify an error message, and then go to the MUSCLE3 issues on GitHub to see if the problem has been reported already, and if there’s a work-around.

If not, please make a new issue with a description of the problem (preferably mention the error in the issue title, so that others can find it), and attach a log of the build. You can make a build log using:

~/muscle3_source/muscle3-0.5.0$ make distclean
~/muscle3_source/muscle3-0.5.0$ make >make.log 2>&1

This will produce a file named make.log with the build output in it. To attach it to a GitHub issue, drag it into the text box from your file manager.

Installing libmuscle C++

Once MUSCLE3 has been compiled, we need to install it. We recommend installing it into a subdirectory of your home directory for now, as opposed to /usr/local/bin or something similar (although /opt/muscle3 would be okay), since there is no uninstall command yet that will cleanly remove it. That goes like this:

~/muscle3_source/muscle3-0.5.0$ PREFIX=~/muscle3 make install

This command will install the C++ version of MUSCLE3 into the directory specified by PREFIX, in this case the muscle3 directory in your home directory.

From this point on, the source directory is no longer needed. If you don’t want to play with the examples (in docs/source/examples/cpp) then you can remove it if you want.

Compiling and linking with libmuscle C++

Once libmuscle is installed, you will have to add some code to your model to talk to libmuscle, or you can write a model component from scratch. Examples of how to do that are in the C++ section of this manual. In order to compile and link your code with libmuscle, you have to adjust the compilation and linking commands a bit though.

When compiling and linking, the compiler needs to be able to find the libmuscle headers and libraries. This is done in three steps:

  1. Make libmuscle available in the environment

  2. Tell the compiler which headers to include

  3. Tell the linker which libraries to link against

Making libmuscle available

The MUSCLE3 installation comes with a file that you can source into your shell, which will set some environment variables needed to compile, link and run with libmuscle. Activate it like this:

$ . <PREFIX>/bin/muscle3.env

Compiling with libmuscle - Manually

Next, when compiling, the compiler needs to be able to find the libmuscle headers. This can be done either directly, or using PkgConfig. To directly point to the headers, use

-I${MUSCLE3_HOME}/include

to your compiler command line. MUSCLE3_HOME is set by muscle3.env, and points to the directory where MUSCLE3 is installed. If your submodel uses MPI, then you must add

-DMUSCLE_ENABLE_MPI

as well to make the MPI-specific parts of the libmuscle API available, and of course remember to use mpic++ or mpicxx to compile.

Compiling with libmuscle - PkgConfig

You can also use PkgConfig with appropriate package names to obtain the right compile options. For non-MPI code, use

pkg-config --cflags ymmsl libmuscle

to get the flags, or use your build system’s PkgConfig support. For MPI code, you should use

pkg-config --cflags ymmsl libmuscle_mpi

instead.

Linking with libmuscle - Manually

When linking, the linker needs to be told where to find the ymmsl and libmuscle libraries, and that it should link with them. That’s done by adding

-L${MUSCLE3_HOME}/lib -lymmsl -lmuscle

to the command line, or for MPI implementations:

-L${MUSCLE3_HOME}/lib -lymmsl -lmuscle_mpi

You’re most likely linking dynamically, but libmuscle does install static libraries in case you need them. If you link statically, then you must add the -pthread option, as libmuscle uses background threads for communication.

Linking with libmuscle - PkgConfig

To link a non-MPI component with libmuscle using PkgConfig, use

pkg-config --libs ymmsl libmuscle

to get the flogs, or use your build system’s PkgConfig support with package names ymmsl and libmuscle. For MPI code, you should use

pkg-config --libs ymmsl libmuscle_mpi

instead.

Running with libmuscle

There’s one more thing: the directory that you’ve installed MUSCLE into is probably not in your system’s library search path, and as a result the dynamic linker won’t be able to find the libraries when you run your program. In order to fix this, LD_LIBRARY_PATH must be set, which you can do either by sourcing the muscle3.env file as described above, or with the following command:

~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PREFIX>/lib

If you have just installed MUSCLE3, then the above bits are currently on your screen, so you can just copy-paste them from there.

Intel C++ compiler

Compiling MUSCLE3 with the Intel compiler is currently not supported. Fortunately, in a typical multiscale simulation, only a small amount of time is spent communicating through MUSCLE3. Your submodels will spend most of their time either computing or waiting. Therefore, it helps to compile the submodels with the Intel compiler for some extra performance, but using the Intel compiler for MUSCLE3 doesn’t add much anyway.

To compile your submodels with the Intel compiler, first use the GNU compiler to install MUSCLE3. Then, switch to the Intel compiler, and use it to compile and link it to MUSCLE3 as described above. The compilers are link-compatible, so this should work. (See below if your model is written in Fortran, you need to do it slightly differently there.)

Intel has recently made it a bit easier to install their proprietary compilers, which should allow us to integrated it into the Continuous Integration system. That should enable support for it in MUSCLE3, and since many people want to use the Intel compilers (especially for compiling Fortran codes) we plan to do so in a future release. If this interests you, please add a comment to the issue on Intel compiler support so we know.

Fortran

Once libmuscle is installed, you will have to add some code to your model to talk to libmuscle, or you can write a component from scratch. Examples of how to do that are in the Fortran section of this manual. In order to compile and link your code with libmuscle, you have to adjust the compilation and linking commands a bit though.

The Fortran bindings for libmuscle are a wrapper around the C++ implementation. They will be built automatically if you follow the instructions above for building and installing libmuscle C++, at least if you have a Fortran compiler available. The instructions for Fortran are almost the same as for C++, but we need a couple extra libraries.

Compiling and linking with libmuscle

When compiling and linking, the compiler needs to be able to find the libmuscle headers and libraries. This is done in three steps:

  1. Make libmuscle available in the environment

  2. Tell the compiler which headers to include

  3. Tell the linker which libraries to link against

Making libmuscle available

The MUSCLE3 installation comes with a file that you can source into your shell, which will set some environment variables needed to compile, link and run with libmuscle. Activate it like this:

$ . <PREFIX>/bin/muscle3.env

Compiling with libmuscle - Manually

Next, when compiling, the compiler needs to be able to find the libmuscle modules. This can be done either directly, or using PkgConfig. To directly point to the modules, use

-I${MUSCLE3_HOME}/include

to your compiler command line. MUSCLE3_HOME is set by muscle3.env, and points to the directory where MUSCLE3 is installed. In Fortran, there’s a separate module for MPI support, so there’s no need for a -D option. You do need to use mpifort or similar to compile.

Compiling with libmuscle - PkgConfig

You can also use PkgConfig with appropriate package names to obtain the right compile options. For non-MPI code, use

pkg-config --cflags ymmsl_fortran libmuscle_fortran

to get the flags, or use your build system’s PkgConfig support. For MPI code, you should use

pkg-config --cflags ymmsl_fortran libmuscle_mpi_fortran

instead.

Linking with libmuscle - Manually

When linking, the linker needs to be told where to find the ymmsl and libmuscle libraries, and that it should link with them. That’s done by adding

-L${MUSCLE3_HOME}/lib -lymmsl_fortran -lmuscle_fortran -lymmsl -lmuscle

to the command line, or for MPI implementations:

-L${MUSCLE3_HOME}/lib -lymmsl_fortran -lmuscle_mpi_fortran -lymmsl -lmuscle_mpi

You’re most likely linking dynamically, but libmuscle does install static libraries in case you need them. If you link statically, then you must add the -pthread option, as libmuscle uses background threads for communication.

Linking with libmuscle - PkgConfig

To link a non-MPI component with libmuscle using PkgConfig, use

pkg-config --libs ymmsl_fortran libmuscle_fortran

to get the flogs, or use your build system’s PkgConfig support with package names ymmsl and libmuscle. For MPI code, you should use

pkg-config --libs ymmsl_fortran libmuscle_mpi_fortran

instead.

Running with libmuscle

There’s one more thing: the directory that you’ve installed MUSCLE into is probably not in your system’s library search path, and as a result the dynamic linker won’t be able to find the libraries when you run your program. In order to fix this, LD_LIBRARY_PATH must be set, which you can do either by sourcing the muscle3.env file as described above, or with the following command:

~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PREFIX>/lib

If you have just installed MUSCLE3, then the above bits are currently on your screen, so you can just copy-paste them from there.

Intel Fortran compiler

MUSCLE3 cannot currently be compiled with the Intel toolchain (see above under C++ for details). You can however compile your submodel with the Intel compiler and link it to the GNU-compiled MUSCLE3 library, because the compilers are almost compatible. The one issue is that the .mod files created by the GNU compiler (and installed when you install libmuscle) cannot be read by the Intel compiler.

To solve this, you need to use the corresponding .f90 files instead. These are installed by in <PREFIX>/include, and are called libmuscle.f90, ymmsl.f90, and libmuscle_mpi.f90. You compile these as you would any other source file in your submodel, and then link them with rest of the submodel and the shared library as described above.

Intel has recently made it a bit easier to install their proprietary compilers, which should allow us to integrated it into the Continuous Integration system. That should enable support for it in MUSCLE3, and since many people want to use the Intel compilers (especially for compiling Fortran codes) we plan to do so in a future release. If this interests you, please add a comment to the issue on Intel compiler support so we know.