Installing

As described before, MUSCLE 3 consists of several components: libmuscle, the YMMSL Python library, and the MUSCLE Manager. Furthermore, libmuscle currently has a Python and a C++ version.

Python

Installing MUSCLE 3 on Python will install all the Python-based components of the system, i.e. the Python version of libmuscle, the YMMSL Python library, and the MUSCLE Manager. This requires at least Python 3.5.

MUSCLE 3 is on PyPI as an ordinary Python package, so it can be installed via Pip in the usual way. It’s normally a good idea to make a virtual environment (virtualenv), if you don’t yet have one:

~$ python3 -m venv muscle3_venv
~$ . muscle3_venv/bin/activate
(muscle3_venv)~$ pip install -U pip setuptools wheel
(muscle3_venv)~$ pip install muscle3

This will create a Python virtualenv in a directory named muscle3_venv in your home directory, and then activate it. This means that when you run Python, it will use the version of Python in the virtual environment, and see the packages you have installed there.

Next, we upgrade pip, the Python package installer (most systems have an old version, and old versions sometimes give problems), setuptools (same thing) and we install wheel (which can cause packages to fail to install if it’s not there).

Having made a good environment, we can then install MUSCLE 3 inside of it. Once that’s done, you can use MUSCLE 3 whenever you have the virtualenv activated. This will also install the Python YMMSL library, and any required dependencies.

You can also install MUSCLE 3 without a virtualenv if your system allows that. The advantage of virtual environments is that you can keep different programs separate, and reduce the chance of library version mismatches. On the other hand, not having to activate the virtual environment saves you a step. If you get any error messages, try upgrading pip, setuptools and wheel as shown above, and then try again.

If you want to install the Python YMMSL library without installing MUSCLE 3, then you can use

~$ pip3 install ymmsl

C++

To work with MUSCLE 3 from C++, you need to install the C++ version of libmuscle. Currently, that means building it from source. This is a bit more involved than installing the Python version, but comparable to (and maybe slightly easier than) installing most C++ libraries.

Prerequisites

To build libmuscle, we’re going to need some tools. In particular, we need a C++ compiler and GNU make. MUSCLE 3 uses C++14, so you need at least g++ 4.9.3. Clang is expected to work, but that’s not been tested. Using the Intel toolchain currently does not work, due to issues with building gRPC with it (but see below for information about building submodels with the Intel tools). Other compilers have not been tested. If you want to try, go right ahead, we’d love to have feedback on this. Building has been tested with gmake 3.82 and 4.1.

If you’re doing C++ development on a reasonably modern Linux, then you probably already have a suitable compiler installed. If not, on a Debian (or Ubuntu) based system, sudo apt-get install build-essential cmake gfortran pkg-config wget should get you set up. On a cluster, there is usually a module load g++ or similar command available that sets you up with g++ and associated tools, and similar for a Fortran compiler. The exact command will vary from machine to machine, so consult the documentation for your cluster and/or ask the helpdesk.

If your submodels use MPI, then you’ll need an MPI library. Libmuscle has been tested with OpenMPI on Ubuntu, but should work with other MPI implementations (this being the point of the MPI standard). On Debian/Ubuntu, sudo apt-get install libopenmpi-dev will install the OpenMPI development files needed to compile libmuscle C++ with MPI support. On a cluster, there is probably a module load openmpi or similar command to make MPI available.

(No) Dependencies

MUSCLE 3 uses several third-party libraries, which need to be available when it is built. If you have them available, they should be detected automatically; if not, MUSCLE 3 will download and install them automatically.

The dependencies are:

  • c-ares 1.11.0 or later

  • gRPC 1.24.3

  • MessagePack 3.2.0 or later

  • OpenSSL 1.0.2 or later

  • Protobuf 3.10.0

  • zlib 1.2.x

If your model uses any of these dependencies directly, then it’s best to install that dependency on your system, either via the package manager or from source, and then link both your library and MUSCLE 3 to the dependency. (See below for how to point the build to your installation.) This avoids having two different versions around and active at the same time. Otherwise, it’s easier to rely on the automatic installation. Note that the gRPC and Protobuf dependencies are exact; getting them to install and work correctly on all systems is unfortunately hard enough already if you limit yourself to a single version, so that is what we do.

Downloading MUSCLE 3

With the tools available, we can download and install MUSCLE 3. First, we create a working directory, download MUSCLE 3 into it, then unpack the downloaded archive and enter the main directory:

~$ mkdir muscle3_source
~$ cd muscle3_source
~/muscle3_source$ wget https://github.com/multiscale/muscle3/archive/0.4.0/muscle3-0.4.0.tar.gz
~/muscle3_source$ tar xf muscle3-0.4.0.tar.gz
~/muscle3_source$ cd muscle3-0.4.0

Of course, you can put the source anywhere you like.

Building MUSCLE 3

The basic command for building MUSCLE 3 is:

~/muscle3_source/muscle3-0.4.0$ make

There are a few options that can be added by setting them as environment variables. These are as follows:

MUSCLE_ENABLE_MPI=1

Compile the MPI version of libmuscle as well as the non-MPI version. This requires an MPI library (including development files) to be available, as described above.

NCORES=<n>

Use the given number of cores to compile MUSCLE 3. By default, MUSCLE 3 will use as many cores (threads) as you have. If you want to use fewer, you can set the number here. Using more will not make it go faster, and is not recommended.

CXX=<compiler command>

By default, MUSCLE 3 will try to compile itself using g++. If you want to use a different compiler, then you can set CXX to something else. The MPI version will always be compiled with mpic++.

DOWNLOAD=<download command>

MUSCLE 3 will try to use either wget or curl to download dependencies. This lets you override the command to use, or select one explicitly.

TAR=<tar command>

This overrides the command used to unpack dependencies, which by default is tar.

protobuf_ROOT=<directory>

Also look in the given directory when detecting the ProtoBuf library. This should be the top of the installation tree, so it will have include/, lib/ and bin/ subdirectories.

grpc_ROOT=<directory>

Also look in the given directory when detecting the gRPC library.

msgpack_ROOT=<directory>

Also look in the given directory when detecting the MsgPack library.

googletest_ROOT=<directory>

Also look in the given directory when detecting the GoogleTest library.

c-ares_ROOT=<directory>

Also look in the given directory when detecting the c-ares library.

zlib_ROOT=<directory>

Also look in the given directory when detecting zlib.

openssl_ROOT=<directory>

Also look in the given directory when detecting OpenSSL.

As an example, to build libmuscle with MPI support, and using 2 cores, you would do:

~/muscle3_source/muscle3-0.4.0$ MUSCLE_ENABLE_MPI=1 NCORES=2 make

This will take ten minutes or so (including building the dependencies), depending on the speed of your machine.

Getting help

The plan is for this to always magically work, but operating systems being as diverse as they are (especially on HPC machines), it’s possible that the build will fail. In that case, have a look at the output to see if you can identify an error message, and then go to the MUSCLE 3 issues on GitHub to see if the problem has been reported already, and if there’s a work-around.

If not, please make a new issue with a description of the problem (preferably mention the error in the issue title, so that others can find it), and attach a log of the build. You can make a build log using:

~/muscle3_source/muscle3-0.4.0$ make distclean
~/muscle3_source/muscle3-0.4.0$ make >make.log 2>&1

This will produce a file named make.log with the build output in it. To attach it to a GitHub issue, drag it into the text box from your file manager.

Installing libmuscle C++

Once MUSCLE 3 has been compiled, we need to install it. We recommend installing it into a subdirectory of your home directory for now, as opposed to /usr/local/bin or something similar (although /opt/muscle3 would be okay), since there is no uninstall command yet that will cleanly remove it. That goes like this:

~/muscle3_source/muscle3-0.4.0$ PREFIX=~/muscle3 make install

This command will install the C++ version of MUSCLE 3 into the directory specified by PREFIX, in this case the muscle3 directory in your home directory.

From this point on, the source directory is no longer needed. If you don’t want to play with the examples (in docs/source/examples/cpp) then you can remove it if you want.

Compiling and linking with libmuscle C++

Once libmuscle is installed, you will have to add some code to your model to talk to libmuscle, or you can write a compute element from scratch. Examples of how to do that are in the C++ section of this manual. In order to compile and link your code with libmuscle, you have to adjust the compilation and linking commands a bit though.

When compiling, the compiler needs to be able to find the MUSCLE 3 headers. You can point it to them by adding

-I<PREFIX>/include

to your compiler command line, where <PREFIX> is where you installed it. If your submodel uses MPI, then you must add

-DMUSCLE_ENABLE_MPI

as well to make the MPI-specific parts of the libmuscle API available, and of course remember to use mpic++ or mpicxx to compile.

When linking, the linker needs to be told where to find the ymmsl and libmuscle libraries, and that it should link with them. That’s done by adding

-L<PREFIX>/lib -lymmsl -lmuscle

to the command line, or for MPI compute elements:

-L<PREFIX>/lib -lymmsl -lmuscle_mpi

You’re most likely linking dynamically, but libmuscle does install static libraries in case you need them. If you link statically, then you must add the -pthread option, as libmuscle uses background threads for communication.

MUSCLE 3 also supports pkg-config. To use pkg-config, add <PREFIX>/lib/pkgconfig to your PKG_CONFIG_PATH and use the module names ymmsl and either libmuscle or libmuscle_mpi.

There’s one more thing: the directory that you’ve installed MUSCLE into is probably not in your system’s library search path, and as a result the dynamic linker won’t be able to find the libraries when you run your program. In order to fix this, LD_LIBRARY_PATH must be set, which you can do with the following command:

~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PREFIX>/lib

If you have just installed MUSCLE 3, then the above bits are currently on your screen, with <PREFIX> filled out already, so you can just copy-paste them from there.

Intel C++ compiler

Compiling MUSCLE 3 with the Intel compiler is currently not supported. One issue here is that gRPC does not compile with the Intel compiler, and it’s a required dependency. If you’re willing and able to do some manual work, you may be able to manually install gRPC using the GNU toolchain, and then compile MUSCLE 3 with the Intel compiler, which may or may not work (if you encounter any problems, please make an issue on GitHub).

However, in a typical multiscale simulation, only a small amount of time is spent communicating through MUSCLE 3. Your submodels will spend most of their time either computing or waiting. Therefore, it helps to compile the submodels with the Intel compiler for extra performance, but using the Intel compiler for MUSCLE 3 doesn’t add much anyway.

To compile your submodels with the Intel compiler, first use the GNU compiler to install MUSCLE 3. Then, switch to the Intel compiler, and use it to compile and link it to MUSCLE 3 as described above. The compilers are link-compatible, so this should work. (See below if your model is written in Fortran, you need to do it slightly differently there.)

Note that since the Intel toolchain is proprietary, continuous testing cannot be done easily, and support for it is a bit experimental still. Nevertheless, the Intel tools are widely used in computational science, so we very much want them to work. If you encounter a problem, please make an issue on GitHub and we’ll try to figure out a solution.

Fortran

The Fortran bindings for libmuscle are a wrapper around the C++ implementation. They will be built automatically if you follow the instructions above for building and installing libmuscle C++.

Compiling and linking with libmuscle Fortran

Once libmuscle is installed, you will have to add some code to your model to talk to libmuscle, or you can write a compute element from scratch. Examples of how to do that are in the Fortran section of this manual. In order to compile and link your code with libmuscle, you have to adjust the compilation and linking commands a bit though.

When compiling, the compiler needs to be able to find the MUSCLE 3 modules. You can point it to them by adding

-I<PREFIX>/include

to your compiler command line, where <PREFIX> is where you installed it.

When linking, the linker needs to be told where to find the ymmsl and libmuscle libraries and their Fortran wrappers, and that it should link with them. That’s done by adding

-L<PREFIX>/lib -lymmsl_fortran -lmuscle_fortran -lymmsl -lmuscle

If your compute element uses MPI, then you need to link to the MPI versions of the libmuscle library instead:

-L<PREFIX>/lib -lymmsl_fortran -lmuscle_mpi_fortran -lymmsl -lmuscle_mpi

MUSCLE 3 also supports pkg-config. To use pkg-config, add <PREFIX>/lib/pkgconfig to your PKG_CONFIG_PATH and use the module names ymmsl_fortran and either libmuscle_fortran or libmuscle_mpi_fortran.

There’s one more thing: the directory that you’ve installed MUSCLE into is probably not in your system’s library search path, and as a result the dynamic linker won’t be able to find the libraries when you run your program. In order to fix this, LD_LIBRARY_PATH must be set, which you can do with the following command:

~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PREFIX>/lib

If you have just installed MUSCLE 3, then the above bits are currently on your screen, with <PREFIX> filled out already, so you can just copy-paste them from there.

Intel Fortran compiler

MUSCLE 3 cannot currently be compiled with the Intel toolchain (see above under C++ for details). You can however compile your submodel with the Intel compiler and link it to the GNU-compiled MUSCLE 3 library, because the compilers are almost compatible. The one issue is that the .mod files created by the GNU compiler (and installed when you install libmuscle) cannot be read by the Intel compiler.

To solve this, you need to use the corresponding .f03 files instead. These are installed by in <PREFIX>/include, and are called libmuscle.f03, ymmsl.f03, and libmuscle_mpi.f03. You compile these as you would any other source file in your submodel, and then link them with rest of the submodel and the shared library as described above.

Note that since the Intel toolchain is proprietary, continuous testing cannot be done easily, and support for it is a bit experimental still. Nevertheless, the Intel tools are widely used in computational science, so we very much want them to work. If you encounter a problem, please make an issue on GitHub and we’ll try to figure out a solution.