How to run and develop tests for the Datamaster

General Framework

The tests for the datamaster are designed for use during development of datamaster features and to run in Jenkins jobs. The tests need a datamaster and a timing receiver. Some tests start patterns on the datamaster and snoop the timing messages on the receiver. These messages are analyzed for the correct value and count of the parameter field or other fields. libcarpedm and dm-cmd, dm-sched are used from the current branch targeting a possibly remote datamaster. Snooping the timing messages uses saft-ctl and for remote timing receivers ssh.

The Jenkins jobs run all tests. To reproduce failures or to run single tests during feature development use the options of pytest.

Prerequisites

Prerequisites: Python 3.6.9 or higher, pytest 6.1.2 or higher.

Running the tests requires an initialized clone of bel_projects. This means: run make in the root folder of bel_projects and make sure that PATH contains the lm32 toolchain. For details see the Readme of bel_projects. The branches dm-fallout-tests, dm-dryrun-2024, dm-new-logging-jul24 contain a current version (2024-08-28) of the tests. All branches which are created from these branches will also contain the tests. The branch fallout contains an older version.

In addition, add your ssh key to the host (default: fel0069.acc.gsi.de) of the timing receiver. You can use ssh-copy-id for this. This is needed because ssh is involved in the saft-ctl command to snoop timing messages.

Etherbone tools: eb-reset should be installed.

scheduleCompare should be installed. See below.

BOOST libraries: the three libraries libboost_graph, libboost_regex, libboost_serialization are needed for libcarpedm. To check the BOOST version, use modules/ftm/tests/tools/getBoostVersion.sh. This script contains its own description.

Run Tests with pytest

The folder modules/ftm/tests has a Makefile. The important targets are: all, remote, prepare. The target all runs tests with pytest against a datamaster $DM and a timing receiver $TR0. Usually these are local. Target remote runs tests with pytest against a datamaster tcp/fel0069.acc.gsi.de and a timing receiver dev/wbm1 on fel0069.acc.gsi.de. Target prepare: make prepare compiles libcarpedm and dm-cmd, dm-sched and in addition singleEdgeTest, scheduleCompare in the current branch. These are prerequisites for the tests. make prepare-all does the same and additionally installs the binaries in $HOME/.local. Use sudo make prepare-all to install the binaries for all users on the machine.

Important: The tests need exclusive access to the datamaster and the timing receiver. Otherwise the schedules and timing messages may be not the ones to test.

Examples:

OPTIONS='--runslow' make

Run all tests against the local datamaster, even those marked with --runslow. These test take longer than usual tests.

OPTIONS='--runslow -k test_threadsStartStop' make

Run tests with test_threadsStartStop in the file name, the class name or the test method name. In this case it is a file name.

OPTIONS='-k zzz' make remote

Run a simple test which uses a schedule sending a timing message every second. Because of target 'remote' this test runs against fel0069.acc.gsi.de. Change the environment variables DATAMASTER and SNOOP for the target test in Makefile to use a different datamaster and timing receiver. Example:

DATAMASTER=host1.acc.gsi.de SNOOP="ssh root@scuxlnnnn.acc.gsi.de 'saft-ctl trY -xv snoop 0 0 0'" OPTIONS='-k zzz' make test

The value of OPTIONS contains arguments which are shipped to pytest. See pytest --help for more options.

Prepare Host and Datamaster to run Tests

In folder modules/ftm/tests of your bel_projects repository run the following:

1. make prepare
   Build DM tools, libcarpedm, and the test binaries to ../bin and ../lib
2. ./tools/datamasterInit ftm.bin {8,32}
   Build datamaster lm32-firmware and upload this to the datamaster.
   Example: ./tools/datamasterInit fel0069 ftm.bin 32
   Builds the lm32-firmware for 32 threads and uploads it to fel0069.acc.gsi.de (all CPUs).

Now you can run the tests.

Initialize and Check Status of a Datamaster

There are two bash scripts to initialize and to test a datamaster in a development environment. Development environment means: the datamaster runs in a separate network (no BOOTP, no local PTP master).

1. tools/datamasterInit.sh
   1. This script checks the existence of the two wishbone devices needed.
   2. Then the PTP master and slave are synchronized with the host.
   3. Check that saftbus service runs.
   4. Build the lm32 firmware and upload this to all CPUs. Arguments of the script define which lm32 firmware to use.
   5. Run a simple test.
      This script is tested on ACOPC042 and fel0069.acc.gsi.de. On other hosts the script needs improvements.
2. tools/datamasterCheck.sh
   1. This script checks the existence of the two wishbone devices needed.
   2. With eb-mon several values of the two wishbone devices are shown.
   3. Check that saftbus runs.
   4. Run a simple test.
      This script is tested on ACOPC042 and fel0069.acc.gsi.de. On other hosts the script needs improvements.

Examples, run in folder modules/ftm/tests. Other folders need adoptions.

./tools/datamasterInit.sh ACOPC042 ftm.bin 8
./tools/datamasterInit.sh fel0069 ftm.bin 32
./tools/datamasterCheck.sh ACOPC042
./tools/datamasterCheck.sh fel0069.acc
./tools/datamasterCheck.sh remote fel0069

Arguments of datamasterInit are the host, name of the lm32 binary, number of threads. Arguments of datamasterCheck are keyword 'remote' (optionally) and the datamaster host.

Develop Tests with Python and pytest

Naming convention

1. Files with unit tests have names starting with test_ and ending with .py.
2. Unit tests are in a test method. The name starts with test and is member of a class derived from class DmTestbench in module dm_testbench.

Module dm_testbench

The module dm_testbench contains methods to

1. add schedules
2. start pattern
3. run dm-cmd and dm-sched
4. snoop timing messages into csv files parallel to executing commands
5. analyse csv files with timing messages
6. analyse the output of dm-cmd $DM -v for number of messages per thread

The detailed documentation of the module is in datamaster-tests.pdf, last section.

Run Test on ASL cluster

Preparation

Use the script for the Jenkins test jobs to build all prerequisites.

git clone https://github.com/GSI-CS-CO/bel_projects.git bel_projects
cd bel_projects
./modules/ftm/tests/tools/jenkinsBuild.sh

modules/ftm/tests/tools/jenkinsBuild.sh should be started from somewhere inside the git repository. It sets the variable WORKSPACE to the root of the repository, if WORKSPACE is not set before.

Check that the following is up to date:

1. the binary scheduleCompare at /common/usr/timing/bin
2. the path to the timing runtime in modules/ftm/ftmx86/Makefile

Check that ssh root@fel0069.acc.gsi.de 'uname -a' works. Add your ssh key to the authorized keys of root at fel0069, i.e. with ssh-copy-id.

Run Single Tests

Use make with the target remote to run the tests against the datamaster on fel0069.acc.gsi.de. See section Run Tests with pytest Example:

OPTIONS='--runslow -k test_threadsStartStop' make remote