Introduction to Cactus
Cactus is an open source, 'problem solving environment' which started life at the Max Planck Institute for Gravitational Physics, before being further developed by collaboration between various institutions. Originally designed for working on numerical relativity problems, cactus has since expanded into a more general tool for scientists and engineers. It currently supports C, C++, F70 and F90. The name 'Cactus' is an attempt to communicate its modular nature, in which users develop their own 'thorns' (modules), which are connected by the 'flesh' (core). Thorns range from the specific, including CFD and numerical relativity applications, to the general, such as parallelisation and mesh refinement tools.
Cactus is called through a simple interface, and aims to provide improvements to both the convenience and reproducibility of numerical work.
The flesh of Cactus allows for utilising thorns of different languages simultaneously and interchangeably, meaning that existing code need not be rewritten before being used. It also negates the effects of machine architecture, meaning that code which runs on a laptop can be seamless run on a supercomputer.
Whilst pre-existing thorns can automate common requirements, e.g. mesh refinement, numerical techniques and parallelisation. In addition, there exists a thorn for a web interface from which simulations can be viewed, paused, and altered/updated in real time.
Once Cactus is installed, two requirements must be satisfied to utilise Cactus to run a script. First, a script needs to be turned into a thorn, then parameter files need to be created to describe the I/O of parameters from, and between, thorns.
Getting started with Cactus
In the following examples, the Cactus directory and all other relevant directories and files are assumed be stored on the desktop or in the root Cactus directory, as is the case for the provided Ubuntu virtual machine. All examples and solutions can be found on the desktop or in this zip.
Cactus' source code is downloaded using Git and Subversion through the use of the Perl script GetComponents provided by the developers of Cactus. GetComponents can be easily downloaded and executed by using the
wget command to download the script and
chmod to make the script executable in the following way,
$ cd ~/Desktop
$ wget https://raw.github.com/gridaphobe/CRL/Cactus_4.3.0/GetComponents
$ chmod a+x GetComponents
To then download Cactus, GetComponents is run with the additional argument of the local path or URL to a thornfile,
$ ./GetComponents Thorn\ Lists/JustCactus.th
Running this script will download the Cactus 'flesh' and any additional thorns that were specified in the thornfile; if they were specified. A thornfile is written in Component Retrival Language (CRL) and the user is referred to the wiki for CRL if they wish to create their own thornfile from scratch.
GetComponents will create a directory containing all of the source code, documentation, a makefile and utilities needed to use Cactus. The core of Cactus is stored in the
src directory and any thorns downloaded will be stored in the
arrangements directory. Each individual thorn belongs to an arrangement, where arrangements are created to keep thorns together for a certain physics problem, purpose, simulation toolkit or author. For this workshop, it's suggested that all created thorns are kept in their own arrangement, for example
Task 1 - creating a new thorn
The first task of the workshop is to create a simple Hello World thorn. To create a new thorn, we first need to download Cactus using the thornlist
JustCactus.th located on the desktop in the
Thorn Lists directory. This will download the core of Cactus and nothing else. Once Cactus is installed, a new thorn is created by invoking the makefile located in the root directory of Cactus with the argument newthorn. At any time, the available arguments for the makefile can be viewed by typing
make into the terminal and pressing tab twice. You can also view all of the possible commands by typing
$ cd ~/Desktop
$ ./GetComponents ThornLists/JustCactus.th
$ cd Cactus
$ make newthorn
You will then be prompted to give your new thorn a name. For this task, name the thorn
HelloWorld or similar. When prompted which arrangement to place the thorn in, create a new one such as
FEEG6003. The next three prompts will ask for the author's name and email of the thorn, and then will prompt for any additional authors. The final prompt is the type of copyright license to be used for the torn. The GNU General Purpose License 2.0 is fine to use.
Once the prompts are filled in, the makefile will create a template thorn with all the necessary files and directories. To navigate and view the new thorn using the terminal, type,
$ cd arrangements/FEEG6003/HelloWorld
The main files and directories for this workshop to pay attention to are,
configuration.ccl: this defines any optional configuration options for a thorn.
interface.ccl: this defines the implementation the thorn provides and the variables which the thorn requires.
param.ccl: this defines all the parameters used to control the thorn, as well as definitions of their visibility to other thorns and implementations.
schedule.ccl: this defines which functions in the source code are called by the thorn and when they are used.
src/: the directory where all the source code for the thorn is stored.
src/make.code.defn: a definition file required for Cactus to know which source files to use to compile the thorn.
All source code written for the thorn is stored in the
src directory. The file
make.code.defn located in the
src directory is used by Cactus to know which files to use to compile the thorn. The
.ccl files are written in Cactus Configuration Language (CCL). The reader is referred to the Cactus User Guide, page C5, for a complete description of the syntax of CCL.
Task 2 - hello world!
Whilst writing a Hello World program in C, C++ or FORTRAN (or really any other language) is simple, writing a Hello World program in Cactus requires a bit more effort. This task will guide you through the steps required to print 'Hello World!' to the Cactus output.
In C and FORTRAN, a simple Hello World program could look like,
WRITE(*,*) "Hello World!"
END PROGRAM HelloWorld
However, in Cactus it's not as simple as putting a print function in your code. Cactus requires each thorn to be a function, or subroutine for FORTRAN, which returns no value. We also can't print to the terminal by using the
WRITE commands. Instead, we have to use the inbuilt Cactus function
CCTK_INFO which is used to print extra information to the terminal whilst Cactus is running.
Writing the source code
Now to begin writing the Cactus thorn! Create a new file in the
src directory named
HelloWorld.c. At the top of each source code, whether written in C, C++ or FORTRAN, we need to include the relevant C style headers for the Cactus variables and functions used. Every thorn requires the
cctk.h header file, which contains the Cactus infrastructure functions. We will also be using the
cctk_Arguments.h header file, so our function can take in arguments from Cactus using
CCTK_ARGUMENTS. To declare these arguments, we include the
DECLARE_CCTK_ARGUMENTS at the top with all of the other variable declarations. Thus, a simple Hello World program, written in C and FORTRAN, for Cactus looks like,
/* declare the Cactus arguments */
/* print to the Cactus output */
! declare the Cactus arguments
! print to the Cactus output
CALL CCTK_INFO("Hello World!")
END SUBROUTINE HelloWorld
Cactus configuration files
With the source code written, we now need to add an appropriate entry to
make.code.defn so Cactus knows to which source files need to be compiled. Open the file and add to the
SRCS line the name of the file containing the source code. If we had multiple source files, we would write each source file separated by a space. As we have not placed any source files in a sub directory in
src, we can leave the
SUBDIRS line blank. The completed
make.code.defn file should look like,
# Main make.code.defn file for thorn HelloWorld
# Source files in this directory
SRCS = HelloWorld.c
# Subdirectories containing source files
However, we still haven't written a functioning Hello World program for Cactus. To complete the program, we need to fill in the
configuration.ccl can be ignored as our simple Hello World program is not using any external parameters.
schedule.ccl, we need to tell when Cactus should call our function and the language our function is written in. The
schedule.ccl file should look like,
# Schedule definitions for thorn HelloWorld
schedule HelloWorld at CCTK_EVOL
} "Print Hello World message to the screen"
schedule HelloWorld at CCTK_EVOL tells Cactus to call our function
HelloWorld at each Cactus evolution (time)step.
LANG:C tells Cactus that our function
HelloWorld is written in C and the string at the end
"Print Hello World message to the screen" provides a description of what we have scheduled. All three of these lines are required in this exact format for Cactus to know when to run our function.
Finally, we need to edit the
interface.ccl file to implement our thorn. This file should look like,
# Interface definition for thorn HelloWorld
implements: tells Cactus what our implementation is named and the line
inherits: tells Cactus which other implementation our own implementation requires. For example, if we wanted to use variables or functions from the
grid thorn, we would need to include
inhereits: grid to tell Cactus that we wish to use that thorn's variables and functions.
Building and running our program
Now that all the required files are completed, we can finally build our Hello World Cactus program. To do this, we will be using the makefile located in the root Cactus directory. The first step is to navigate to the root Cactus directory and create a configuration file,
$ cd ~/Desktop/Cactus
$ make HelloWorld-config
The makefile file will create a configuration with the argument
-config is vital as this informs the makefile you are creating a configuration. The configuration command will prompt to confirm that you want to make the configuration. It will then check if the machine has all the required dependencies to properly build all of the currently installed thorns in the
arrangements directory. Once this has finished running, the configuration will be stored in the directory
configs/ProgramName-config. To now build a Cactus program, we now type into the terminal,
You will be prompted with a list of all currently installed thorns and given the option yes or no to edit the list of which thorns to compile. If you select no, all installed thorns will be compiled. If you select yes, you will be presented a
vi interface to edit this list. Note: if you choose to compile with all the installed thorns, this can significantly increase the time it takes to build the program. For this task, no other thorns should be installed so we can enter no to begin the build process.
Once this build process is complete, the executable for the program is located in the
exe/ directory with the name
cactus_ProgramName. However, one final hurdle is still left. To run our program, we need to provide a parameter file.
By convention, the parameter file should be named
ProgramName.par, however there is no restriction to the name of it. The parameter files requires a list of the thorns which will be used in the program. This is done by adding
ActiveThorns = "ThornName1 ThornName2 ThornName3" to the parameter file. Each activated thorn and the Cactus core has parameters which control the implementations provided by the thorns. To define a variable we use the syntax
implementation::variable=value. For our Hello World program, we can use a simple parameter file
# Parameters for HelloWorld program
ActiveThorns = "HelloWorld"
# cactus core parameters
cactus::cctk_itlast = 10
ActiveThorns = "HelloWorld" is telling Cactus that we want to use the HelloWorld thorn we have just written and built. Then, the line
cactus::cctk_itlast = 10 tells the cactus implementation (which is one of the core Cactus implementations) that we want to do 10 iterations in our program. This will result in 10
With the parameter file complete, we can now finally run our program! Assuming we are in the root Cactus directory, and this is where our parameter file is saved,
$ ./exe/cactus_HelloWorld HelloWorld.par
If all has gone well
INFO (HelloWorld): Hello World! will be printed to the Cactus output 10 times!
Task 3 - using external parameters and printing formatted strings
Now that we know how to write a simple Hello World program, we can start to implement more core Cactus utilities. In this task we will implement the ability to take in parameters from a parameter file and in the next task, we will implement public variables, more Cactus functions and use other thorns in conjunction with our own.
Including external parameters
To include external variables into our thorns, we now need to edit the
params.ccl file and include the
cctk_Parameters.h header file.
For your source code, be sure to add
#include "cctk_Parameters.h" at the top and use the function
DECLARE_CCTK_PARAMETERS to declare the parameters in your function; it's best used above or below your
To include external parameters in a program, we need to add entries for them in
params.ccl. This is done in the following way,
REAL ParameterName "description of the parameter"
*:* :: "description of the allowed values"
The definition starts by declaring the data type of the parameter, in this case
REAL. The different types of parameters are:
Note: there are also Cactus versions of these data types, prefixed with CCTK.
The name of the parameter is next, followed by a string describing the parameter. Inside the curly braces, we have the line
*:* :: "description of the allowed values"
REAL we use the above format, where
min:max allows us to set a minimum and maximum value for the parameter. For a
BOOLEAN we can set allowed parameter values by using a string instead of
min:max. The above syntax needs be used exactly, including the description of the allowed parameter values. The final part of the parameter definition is the value after the closing curly bracket. This sets the default value for the parameter, which is a fall-back for the parameter if no value is set in the parameter file.
Now that the parameters have been defined in
param.ccl and declared using
DECLARE_CCTK_PARAMETERS, we can use these parameters freely in our program without declaring them in the functions we write, as it has already been declared for Cactus in
param.ccl and declared in the function's name space with
Printing formatted strings
To print formatted strings in Cactus, we can no longer use
CCTK_INFO. Instead, we need to add another header file
cctk_Functions.h to allow us to use more Cactus functions. For C we have to use the function
CCTK_THORNSTRING in the following way,
CCTK_VInfo(CCTK_THORNSTRING, "X_PLUS = %10.7f", X_P);
For Fortran, we first have to write the formatted string to a variable and then we can call
CCTK_INFO to print this variable.
WRITE(X_M_MSG, '("X_MINUS = ", F10.7)') X_M
For this task, we will be writing a program to solve a quadratic equation given the coefficients of said equation from the parameter file. The two different roots should be printed to the Cactus output. Remember to define the values of the coefficents in the parameter file!
Hint: you should configure
interface.ccl in the same way as the previous task, but changing some values to reflect the task. Also be sure to set
cactus::cctk_itlast to 1, otherwise the equation will be solved multiple times.
Task 4 - more complicated parameters and outputting data
With the example before, we now know how to create and use a simple program which takes in parameters from an external parameter file. Now we will focus on using more complicated parameters, which can be shared between thorns, and outputting data from the thorn to file.
Sharing variables between thorns
If we want to share variables between thorns, we need to include the definitions of the variables in
interface.ccl. Before we would only populate the
inherits fields, but now we meed to populate the
public field will contain any functions and variables which we want available to other thorns which implement our thorn. To define a public variable, we need to use the following form,
<data_type> <group_name> <group_type> <dimensions> <size>
variable1, variable2, variable3
} "description of variables"
For example, if we want to make a 1-dimensional array of a size specified in the parameter file, we could write,
CCTK_REAL group1 type=Array DIM=1 size=(n_points)
} "A 1D array for storing X grid points"
where n_points is a parameter located in
param.ccl. We have to be careful in defining the data type for the variables. In practice is it best to use the
CCTK variable data types, as in the code example above. Note that by defining
size, we are effectively telling Cactus that we wish to allocate memory for our variable.
As a side note, we can also define a variable as being
private instead of
protected variable can be used the thorn it belongs to, as well as any
friends which are defined in
private thorn can only be used in the thorn it belongs to.
Outputting data to file
Now that we have a variable which is pubic to Cactus and other thorns, we can use the thorn
IOASCII to output the variable to file. To do this, we need
IOASCII as an active thorn in our parameter file and then to define values for the parameters.
IOASCII allows us to easily print out the value of a variable to file. If
IOASCII is provided with an array of data, it will print the whole array. If it is provided with a
REAL, then it will just print that value of that real.
Below are some commonly used parameters for
# IOASCII Output
IOASCII::out1D_every = output frequency
IOASCII::out1D_vars = "implementation::variable1 implementatation::variable2"
IOASCII::out1D_dir = "/path/to/output/dir"
IOASCII::out1D_style = "gnuplot f(x)"
The reader is referred to the documentation for
~/Desktop/Documentation/Thorn \Documentation if they wish to read more about the different variables available and their options,
For this task, you will be given a function which solves an ordinary differential equation using the Runge Kutta method. Your task is to finish the provided Cactus configuration and parameter files. To run the code, you will need to use the thorn list provided in
~/Desktop/ThornLists/BasicThorns.th to download the required thorns. To visualise the data, you can use the Python script
plot_RK.py located in the
Scripts directory. Note: because we have added extra thorns to our program, it will now take longer to compile. It takes about 3-5 minutes to compile on a 1.8 GHz MacBook Air.
Task 5 - Remote Access
Now its time to see Cactus in motion, specifically using the WaveToy Demo. Cactus allows for remote viewing of simulations, as well as simulation parameter modification on the fly.
The WaveToy demo is a demo of Cactus' wave equation solving thorns provided by the Cactus developers. The demo can be found here. The WaveToy demo simulates a 3D scalar field produced by two orbiting sources. The solution to the wave equation is found through a finite-differencing algorithm. In our example, we will be using the C version of WaveToy and output the data into Gnuplot ASCII format.
Downloading and running the demo
To download the demo, we checkout Cactus in the same was as before using
GetComponents. For the demo, we will be using this the located in
To run the demo, type in these commands using all of the default options suggested by
make. Note: this can take a long time to compile! It takes around 5 minutes on a MacBook Air with a 1.8 GHz CPU.
$ cd Cactus
$ make WaveDemo-config
$ make WaveDemo
$ ./exe/cactus_WaveDemo ../Parameter\ Files/WaveDemo.par
The output of the file can then either be viewed by using
Gnuplot or by looking at the JPEG slices. The JPEG slices should be located in
WaveDemo/jpeg and are 2D heat maps of the output from Cactus taken at a frequency specified by the parameter file.
Running WaveDemo will print information to the terminal including a line looking something like
Server started on http://feeg6003:****/ where
**** is a 4 digit number. Navigate to this address either by clicking on it, or pasting it into a browser.
Navigate to the 'Cactus Control' tab (username and password: anon), and pause the simulation; note that this will also pause the progress in the terminal.
A steerable parameter is a parameter which we can change on the fly when we use the web iterface. This allows us to tweak simulations whilst they are still running. For example, we can increase the number of iterations we want to run the program for or maybe even change a boundary condition.
Navigating to the 'Parameters' tab show the assigned parameters for each thorn, some of which are 'steerable'. Try altering some of the steerable variables, and use the viewport to see what effect they have on the output.
Now, utilising the fact that you know which parameters are steerable and which aren't, make some of the fixed parameters of WaveDemo into steerable parameters.
Hint: navigate to the folder
~/Desktop/Cactus/arrangements/ and compare the
params.ccl file for various thorns.
Take one of your previously constructed thorns, HelloWorld, Quadratic, or Runge-Kutta, and modify it such that when run, it launches a local server.
Hint: Determine the necessary thorns by comparing the thorns in the parameter file for this task, and then add the necessary thorns to the parameter file and build.
Change the username and password associated with
Hint: The following command allows you to search files for any instance of a particular string.
$ grep -rnw '/path/to/directory/' -e 'string'
Task 6 - MPI
Cactus has thorns available to automate parallelisation, and distribute work across distributed memory systems. The thorns responsible for this can be customised as required for the script to be parallelised.
Now to try and get WaveDemo running on multiple processes. First, we need the thorn
/ExternalLibraries/MPI, which we do using the
GetComponents executable. We also need to add the necessary parameters to the parameter file.
We have to now run the executable with the
Note that, when run in serial, info printed to terminal from the PUGH thorn includes a line to the effect of:
INFO (PUGH): Single processor evolution
MPI thorn is active, this line will be altered to reflect this, and further altered when parallelisation is achieved.
Utilise the web interface to view the output, and note how it changes when run with differing numbers of processors.
Next, utilise the same script as in the extra task in section 5, and try and parallelise this script.
The tasks above allow you to create a cactus thorn from scratch, have it share variables with other thorns, access it through a web interface, and parallelise it. Whilst, in practise these tasks have simplified the problems, in particular with regards to parallelisation. In reality, the behaviour of the PUGH thorn would have to be specified, in order to produce a useful parallel effect. However, in the case of all of these tasks, the essential principles remain the same.
Further reading and details
More complicated and helpful Cactus functions and variables can be found in the Cactus User Guide. The user guide is currently 159 pages long and contains good descriptions of,
- an introduction to Cactus,
- installation, compilation and running of Cactus programs, and,
- thorn writing.
The user guide also contains information abut the infrastructure of Cactus and a large appendix documenting file syntax and utility routines. If you wish to know more Cactus functions, the developers have provided a reference guide which contains a near complete list of all of Cactus' functions and routines.
The tutorials provided by the Cactus development team are a good start if you want to learn how to use Cactus. However, the tutorials are in general quite outdated and need to be updated, but, they are still quite sufficient to learn the basic and intermediate aspects of Cactus.
The Einstein Toolkit
The Einstein Toolkit is a collection of Cactus thorns (a toolkit) used for relativistic astrophysics research. However, instead of using
make to create configurations and build programs like we have used in this blog, the Einstein Toolkit opts to use Sim Factory instead to manage the configuration, building and running of simulations. However, the Einstein Toolkit takes a significant amount of time and a lot of dependencies, hence the reader is linked to this tutorial if they wish to explore and use the Einstein Toolkit.