Cactus - a modular code for computation across different architectures

Introduction to Cactus

Cactus is an open source, 'problem solving environment' which started life at the Max Planck Institute for Gravitational Physics, before being further developed by collaboration between various institutions. Originally designed for working on numerical relativity problems, cactus has since expanded into a more general tool for scientists and engineers. It currently supports C, C++, F70 and F90. The name 'Cactus' is an attempt to communicate its modular nature, in which users develop their own 'thorns' (modules), which are connected by the 'flesh' (core). Thorns range from the specific, including CFD and numerical relativity applications, to the general, such as parallelisation and mesh refinement tools.

Cactus is called through a simple interface, and aims to provide improvements to both the convenience and reproducibility of numerical work.

The flesh of Cactus allows for utilising thorns of different languages simultaneously and interchangeably, meaning that existing code need not be rewritten before being used. It also negates the effects of machine architecture, meaning that code which runs on a laptop can be seamless run on a supercomputer.

Whilst pre-existing thorns can automate common requirements, e.g. mesh refinement, numerical techniques and parallelisation. In addition, there exists a thorn for a web interface from which simulations can be viewed, paused, and altered/updated in real time.

Once Cactus is installed, two requirements must be satisfied to utilise Cactus to run a script. First, a script needs to be turned into a thorn, then parameter files need to be created to describe the I/O of parameters from, and between, thorns.

Getting started with Cactus

In the following examples, the Cactus directory and all other relevant directories and files are assumed be stored on the desktop or in the root Cactus directory, as is the case for the provided Ubuntu virtual machine. All examples and solutions can be found on the desktop or in this zip.

Cactus' source code is downloaded using Git and Subversion through the use of the Perl script GetComponents provided by the developers of Cactus. GetComponents can be easily downloaded and executed by using the wget command to download the script and chmod to make the script executable in the following way,

$ cd ~/Desktop
$ wget https://raw.github.com/gridaphobe/CRL/Cactus_4.3.0/GetComponents
$ chmod a+x GetComponents

To then download Cactus, GetComponents is run with the additional argument of the local path or URL to a thornfile,

$ ./GetComponents Thorn\ Lists/JustCactus.th

Running this script will download the Cactus 'flesh' and any additional thorns that were specified in the thornfile; if they were specified. A thornfile is written in Component Retrival Language (CRL) and the user is referred to the wiki for CRL if they wish to create their own thornfile from scratch.

GetComponents will create a directory containing all of the source code, documentation, a makefile and utilities needed to use Cactus. The core of Cactus is stored in the src directory and any thorns downloaded will be stored in the arrangements directory. Each individual thorn belongs to an arrangement, where arrangements are created to keep thorns together for a certain physics problem, purpose, simulation toolkit or author. For this workshop, it's suggested that all created thorns are kept in their own arrangement, for example arrangements/FEEG6003.

Task 1 - creating a new thorn

The first task of the workshop is to create a simple Hello World thorn. To create a new thorn, we first need to download Cactus using the thornlist JustCactus.th located on the desktop in the Thorn Lists directory. This will download the core of Cactus and nothing else. Once Cactus is installed, a new thorn is created by invoking the makefile located in the root directory of Cactus with the argument newthorn. At any time, the available arguments for the makefile can be viewed by typing make into the terminal and pressing tab twice. You can also view all of the possible commands by typing make help.

$ cd ~/Desktop
$ ./GetComponents ThornLists/JustCactus.th
$ cd Cactus
$ make newthorn

You will then be prompted to give your new thorn a name. For this task, name the thorn HelloWorld or similar. When prompted which arrangement to place the thorn in, create a new one such as FEEG6003. The next three prompts will ask for the author's name and email of the thorn, and then will prompt for any additional authors. The final prompt is the type of copyright license to be used for the torn. The GNU General Purpose License 2.0 is fine to use.

Once the prompts are filled in, the makefile will create a template thorn with all the necessary files and directories. To navigate and view the new thorn using the terminal, type,

$ cd arrangements/FEEG6003/HelloWorld
$ ls

The main files and directories for this workshop to pay attention to are,

configuration.ccl: this defines any optional configuration options for a thorn.
interface.ccl: this defines the implementation the thorn provides and the variables which the thorn requires.
param.ccl: this defines all the parameters used to control the thorn, as well as definitions of their visibility to other thorns and implementations.
schedule.ccl: this defines which functions in the source code are called by the thorn and when they are used.
src/: the directory where all the source code for the thorn is stored.
src/make.code.defn: a definition file required for Cactus to know which source files to use to compile the thorn.

All source code written for the thorn is stored in the src directory. The file make.code.defn located in the src directory is used by Cactus to know which files to use to compile the thorn. The .ccl files are written in Cactus Configuration Language (CCL). The reader is referred to the Cactus User Guide, page C5, for a complete description of the syntax of CCL.

Task 2 - hello world!

Whilst writing a Hello World program in C, C++ or FORTRAN (or really any other language) is simple, writing a Hello World program in Cactus requires a bit more effort. This task will guide you through the steps required to print 'Hello World!' to the Cactus output.

In C and FORTRAN, a simple Hello World program could look like,

#include <stdio.h>

int main(void)
{
  printf("Hello World!");
  return 0;
}

PROGRAM HelloWorld

  IMPLICIT NONE
  WRITE(*,*) "Hello World!"

END PROGRAM HelloWorld

However, in Cactus it's not as simple as putting a print function in your code. Cactus requires each thorn to be a function, or subroutine for FORTRAN, which returns no value. We also can't print to the terminal by using the printf or WRITE commands. Instead, we have to use the inbuilt Cactus function CCTK_INFO which is used to print extra information to the terminal whilst Cactus is running.

Writing the source code

Now to begin writing the Cactus thorn! Create a new file in the src directory named HelloWorld.c. At the top of each source code, whether written in C, C++ or FORTRAN, we need to include the relevant C style headers for the Cactus variables and functions used. Every thorn requires the cctk.h header file, which contains the Cactus infrastructure functions. We will also be using the cctk_Arguments.h header file, so our function can take in arguments from Cactus using CCTK_ARGUMENTS. To declare these arguments, we include the DECLARE_CCTK_ARGUMENTS at the top with all of the other variable declarations. Thus, a simple Hello World program, written in C and FORTRAN, for Cactus looks like,

#include "cctk.h"
#include "cctk_Arguments.h"

void HelloWorld(CCTK_ARGUMENTS)
{
  /* declare the Cactus arguments */
  DECLARE_CCTK_ARGUMENTS
  /* print to the Cactus output */
  CCTK_INFO("Hello World!");
  return;
}

#include "cctk.h"
#include "cctk_Arguments.h"

SUBROUTINE HelloWorld(CCTK_ARGUMENTS)

  ! declare the Cactus arguments
  DECLARE_CCTK_ARGUMENTS
  ! print to the Cactus output
  CALL CCTK_INFO("Hello World!")
  RETURN

END SUBROUTINE HelloWorld

Cactus configuration files

With the source code written, we now need to add an appropriate entry to make.code.defn so Cactus knows to which source files need to be compiled. Open the file and add to the SRCS line the name of the file containing the source code. If we had multiple source files, we would write each source file separated by a space. As we have not placed any source files in a sub directory in src, we can leave the SUBDIRS line blank. The completed make.code.defn file should look like,

# Main make.code.defn file for thorn HelloWorld

# Source files in this directory
SRCS = HelloWorld.c

# Subdirectories containing source files
SUBDIRS =

However, we still haven't written a functioning Hello World program for Cactus. To complete the program, we need to fill in the schedule.ccl and interface.ccl files; params.ccl and configuration.ccl can be ignored as our simple Hello World program is not using any external parameters.

In schedule.ccl, we need to tell when Cactus should call our function and the language our function is written in. The schedule.ccl file should look like,

# Schedule definitions for thorn HelloWorld
schedule HelloWorld at CCTK_EVOL
{
  LANG:C
} "Print Hello World message to the screen"

The line schedule HelloWorld at CCTK_EVOL tells Cactus to call our function HelloWorld at each Cactus evolution (time)step. LANG:C tells Cactus that our function HelloWorld is written in C and the string at the end "Print Hello World message to the screen" provides a description of what we have scheduled. All three of these lines are required in this exact format for Cactus to know when to run our function.

Finally, we need to edit the interface.ccl file to implement our thorn. This file should look like,

# Interface definition for thorn HelloWorld
implements: helloworld
inherits:

The line implements: tells Cactus what our implementation is named and the line inherits: tells Cactus which other implementation our own implementation requires. For example, if we wanted to use variables or functions from the grid thorn, we would need to include inhereits: grid to tell Cactus that we wish to use that thorn's variables and functions.

Building and running our program

Now that all the required files are completed, we can finally build our Hello World Cactus program. To do this, we will be using the makefile located in the root Cactus directory. The first step is to navigate to the root Cactus directory and create a configuration file,

$ cd ~/Desktop/Cactus
$ make HelloWorld-config

The makefile file will create a configuration with the argument ProgramName-config, where -config is vital as this informs the makefile you are creating a configuration. The configuration command will prompt to confirm that you want to make the configuration. It will then check if the machine has all the required dependencies to properly build all of the currently installed thorns in the arrangements directory. Once this has finished running, the configuration will be stored in the directory configs/ProgramName-config. To now build a Cactus program, we now type into the terminal,

$ make HelloWorld

You will be prompted with a list of all currently installed thorns and given the option yes or no to edit the list of which thorns to compile. If you select no, all installed thorns will be compiled. If you select yes, you will be presented a vi interface to edit this list. Note: if you choose to compile with all the installed thorns, this can significantly increase the time it takes to build the program. For this task, no other thorns should be installed so we can enter no to begin the build process.

Once this build process is complete, the executable for the program is located in the exe/ directory with the name cactus_ProgramName. However, one final hurdle is still left. To run our program, we need to provide a parameter file.

By convention, the parameter file should be named ProgramName.par, however there is no restriction to the name of it. The parameter files requires a list of the thorns which will be used in the program. This is done by adding ActiveThorns = "ThornName1 ThornName2 ThornName3" to the parameter file. Each activated thorn and the Cactus core has parameters which control the implementations provided by the thorns. To define a variable we use the syntax implementation::variable=value. For our Hello World program, we can use a simple parameter file HelloWorld.par,

# Parameters for HelloWorld program

ActiveThorns = "HelloWorld"
# cactus core parameters
cactus::cctk_itlast = 10

The line ActiveThorns = "HelloWorld" is telling Cactus that we want to use the HelloWorld thorn we have just written and built. Then, the line cactus::cctk_itlast = 10 tells the cactus implementation (which is one of the core Cactus implementations) that we want to do 10 iterations in our program. This will result in 10 CCTK_EVOL steps.

With the parameter file complete, we can now finally run our program! Assuming we are in the root Cactus directory, and this is where our parameter file is saved,

$ ./exe/cactus_HelloWorld HelloWorld.par

If all has gone well INFO (HelloWorld): Hello World! will be printed to the Cactus output 10 times!

Task 3 - using external parameters and printing formatted strings

Now that we know how to write a simple Hello World program, we can start to implement more core Cactus utilities. In this task we will implement the ability to take in parameters from a parameter file and in the next task, we will implement public variables, more Cactus functions and use other thorns in conjunction with our own.

Including external parameters

To include external variables into our thorns, we now need to edit the params.ccl file and include the cctk_Parameters.h header file.

For your source code, be sure to add #include "cctk_Parameters.h" at the top and use the function DECLARE_CCTK_PARAMETERS to declare the parameters in your function; it's best used above or below your DECLARE_CCTK_ARGUMENTS function.

To include external parameters in a program, we need to add entries for them in params.ccl. This is done in the following way,

REAL ParameterName "description of the parameter"
{
  *:* :: "description of the allowed values"
} 0.0

The definition starts by declaring the data type of the parameter, in this case REAL. The different types of parameters are:

REAL
INT
KEYWORD
STRING
BOOLEAN

Note: there are also Cactus versions of these data types, prefixed with CCTK.

The name of the parameter is next, followed by a string describing the parameter. Inside the curly braces, we have the line

*:* :: "description of the allowed values"

For an INT or REAL we use the above format, where min:max allows us to set a minimum and maximum value for the parameter. For a KEYWORD, STRING OR BOOLEAN we can set allowed parameter values by using a string instead of min:max. The above syntax needs be used exactly, including the description of the allowed parameter values. The final part of the parameter definition is the value after the closing curly bracket. This sets the default value for the parameter, which is a fall-back for the parameter if no value is set in the parameter file.

Now that the parameters have been defined in param.ccl and declared using DECLARE_CCTK_PARAMETERS, we can use these parameters freely in our program without declaring them in the functions we write, as it has already been declared for Cactus in param.ccl and declared in the function's name space with DECLARE_CCTK_PARAMETERS.

Printing formatted strings

To print formatted strings in Cactus, we can no longer use CCTK_INFO. Instead, we need to add another header file cctk_Functions.h to allow us to use more Cactus functions. For C we have to use the function CCTK_VInfo with CCTK_THORNSTRING in the following way,

CCTK_VInfo(CCTK_THORNSTRING, "X_PLUS  = %10.7f", X_P);

For Fortran, we first have to write the formatted string to a variable and then we can call CCTK_INFO to print this variable.

WRITE(X_M_MSG, '("X_MINUS = ", F10.7)') X_M
CALL CCTK_INFO(X_P_MSG)

The task

For this task, we will be writing a program to solve a quadratic equation given the coefficients of said equation from the parameter file. The two different roots should be printed to the Cactus output. Remember to define the values of the coefficents in the parameter file!

Hint: you should configure schedule.ccl and interface.ccl in the same way as the previous task, but changing some values to reflect the task. Also be sure to set cactus::cctk_itlast to 1, otherwise the equation will be solved multiple times.

Task 4 - more complicated parameters and outputting data

With the example before, we now know how to create and use a simple program which takes in parameters from an external parameter file. Now we will focus on using more complicated parameters, which can be shared between thorns, and outputting data from the thorn to file.

Sharing variables between thorns

If we want to share variables between thorns, we need to include the definitions of the variables in interface.ccl. Before we would only populate the implements and inherits fields, but now we meed to populate the public field.

The public field will contain any functions and variables which we want available to other thorns which implement our thorn. To define a public variable, we need to use the following form,

<data_type> <group_name> <group_type> <dimensions> <size>
{
  variable1, variable2, variable3
} "description of variables"

For example, if we want to make a 1-dimensional array of a size specified in the parameter file, we could write,

CCTK_REAL group1 type=Array DIM=1 size=(n_points)
{
  X
} "A 1D array for storing X grid points"

where n_points is a parameter located in param.ccl. We have to be careful in defining the data type for the variables. In practice is it best to use the CCTK variable data types, as in the code example above. Note that by defining data_type, group_name, group_type, dimensions and size, we are effectively telling Cactus that we wish to allocate memory for our variable.

As a side note, we can also define a variable as being protected or private instead of public. A protected variable can be used the thorn it belongs to, as well as any friends which are defined in interface.ccl. A private thorn can only be used in the thorn it belongs to.

Outputting data to file

Now that we have a variable which is pubic to Cactus and other thorns, we can use the thorn IOASCII to output the variable to file. To do this, we need IOASCII as an active thorn in our parameter file and then to define values for the parameters. IOASCII allows us to easily print out the value of a variable to file. If IOASCII is provided with an array of data, it will print the whole array. If it is provided with a REAL, then it will just print that value of that real.

Below are some commonly used parameters for IOASCII.

# IOASCII Output
IOASCII::out1D_every  = output frequency
IOASCII::out1D_vars   = "implementation::variable1 implementatation::variable2"
IOASCII::out1D_dir    = "/path/to/output/dir"
IOASCII::out1D_style  = "gnuplot f(x)"

The reader is referred to the documentation for IOASCII in ~/Desktop/Documentation/Thorn \Documentation if they wish to read more about the different variables available and their options,

The Task

For this task, you will be given a function which solves an ordinary differential equation using the Runge Kutta method. Your task is to finish the provided Cactus configuration and parameter files. To run the code, you will need to use the thorn list provided in ~/Desktop/ThornLists/BasicThorns.th to download the required thorns. To visualise the data, you can use the Python script plot_RK.py located in the Scripts directory. Note: because we have added extra thorns to our program, it will now take longer to compile. It takes about 3-5 minutes to compile on a 1.8 GHz MacBook Air.

Task 5 - Remote Access

Now its time to see Cactus in motion, specifically using the WaveToy Demo. Cactus allows for remote viewing of simulations, as well as simulation parameter modification on the fly.

WaveToy demo

The WaveToy demo is a demo of Cactus' wave equation solving thorns provided by the Cactus developers. The demo can be found here. The WaveToy demo simulates a 3D scalar field produced by two orbiting sources. The solution to the wave equation is found through a finite-differencing algorithm. In our example, we will be using the C version of WaveToy and output the data into Gnuplot ASCII format.

Downloading and running the demo

To download the demo, we checkout Cactus in the same was as before using GetComponents. For the demo, we will be using this the located in ~/Desktop/ThornLists/WaveDemo.th.

To run the demo, type in these commands using all of the default options suggested by make. Note: this can take a long time to compile! It takes around 5 minutes on a MacBook Air with a 1.8 GHz CPU.

$./GetComponents WaveDemo.th
$ cd Cactus
$ make WaveDemo-config
$ make WaveDemo
$ ./exe/cactus_WaveDemo ../Parameter\ Files/WaveDemo.par

The output of the file can then either be viewed by using Gnuplot or by looking at the JPEG slices. The JPEG slices should be located in WaveDemo/jpeg and are 2D heat maps of the output from Cactus taken at a frequency specified by the parameter file.

Running WaveDemo will print information to the terminal including a line looking something like Server started on http://feeg6003:****/ where **** is a 4 digit number. Navigate to this address either by clicking on it, or pasting it into a browser.

Navigate to the 'Cactus Control' tab (username and password: anon), and pause the simulation; note that this will also pause the progress in the terminal.

Steerable Parameters

A steerable parameter is a parameter which we can change on the fly when we use the web iterface. This allows us to tweak simulations whilst they are still running. For example, we can increase the number of iterations we want to run the program for or maybe even change a boundary condition.

Navigating to the 'Parameters' tab show the assigned parameters for each thorn, some of which are 'steerable'. Try altering some of the steerable variables, and use the viewport to see what effect they have on the output.

The Task

Now, utilising the fact that you know which parameters are steerable and which aren't, make some of the fixed parameters of WaveDemo into steerable parameters.

Hint: navigate to the folder ~/Desktop/Cactus/arrangements/ and compare the params.ccl file for various thorns.

Extra Task

Take one of your previously constructed thorns, HelloWorld, Quadratic, or Runge-Kutta, and modify it such that when run, it launches a local server.

Hint: Determine the necessary thorns by comparing the thorns in the parameter file for this task, and then add the necessary thorns to the parameter file and build.

Extra Task

Change the username and password associated with Cactus Control.

Hint: The following command allows you to search files for any instance of a particular string.

$ grep -rnw '/path/to/directory/' -e 'string'

Task 6 - MPI

Cactus has thorns available to automate parallelisation, and distribute work across distributed memory systems. The thorns responsible for this can be customised as required for the script to be parallelised.

The Task

Now to try and get WaveDemo running on multiple processes. First, we need the thorn /ExternalLibraries/MPI, which we do using the GetComponents executable. We also need to add the necessary parameters to the parameter file.

We have to now run the executable with the mpirun command.

Note that, when run in serial, info printed to terminal from the PUGH thorn includes a line to the effect of:

INFO (PUGH): Single processor evolution

When the MPI thorn is active, this line will be altered to reflect this, and further altered when parallelisation is achieved.

Utilise the web interface to view the output, and note how it changes when run with differing numbers of processors.

Extra Task

Next, utilise the same script as in the extra task in section 5, and try and parallelise this script.

Synopsis

The tasks above allow you to create a cactus thorn from scratch, have it share variables with other thorns, access it through a web interface, and parallelise it. Whilst, in practise these tasks have simplified the problems, in particular with regards to parallelisation. In reality, the behaviour of the PUGH thorn would have to be specified, in order to produce a useful parallel effect. However, in the case of all of these tasks, the essential principles remain the same.

Computational Modelling Tools Workshops

Other articles

Cactus - a modular code for computation across different architectures

Introduction to Cactus

Introduction to PyOpenCL

OpenCL

Other articles

Introduction to Cactus

OpenCL

blogroll

social