Simple API For Timing (SaftLib)

Simple API For Timing (SAFT). The design and implementation of SaftLib is a major project.

Introduction

The key features SaftLib have already described in the specification of the timing receiver nodes for FAIR [1]. For a public compilation see here.

Fig1: Software stack of a timing receiver using SaftLib, which provides interfaces to various components of the FPGA. As an exampl, the ECA is shown as a component. For more details see text.

The figure above depicts the SaftLib software stack of a Timing Receiver (TR) with focus on the Event Condition Action unit (ECA). SaftLib implements an interface towards third party user software such as a FESA class. It abstracts the TR by hiding its Wishbone architecture and address layout and provides high level functionality of the TR. SaftLib supports various components of the FPGA (only the ECA is shown here). Despite hiding the complexity of the gateware, one of its prime features is to provide and manage access to the resources of the FPGA, which is vital if a resource cannot be shared.

• Hardware Abstraction: SaftLib auto-detects available resources of the hardware by probing the gateware. Userland applications will work on all types of timing receivers providing relevant Wishbone devices.
• Userland applications must not bypass SaftLib for gaining access to resources that can't be shared. SaftLib provides the features of such resources to applications via Inter Process Communication (IPC).
• Userland applications must not bypass SaftLib for gaining access to resources that are already taken by SaftLib. Otherwise, consistency of states or data cached in SaftLib will break.
• For Wishbone components where SaftLib does not provide a dedicated driver, userland applications may use the Etherbone API.

Interface to ECA

As interface to the ECA SaftLib provides the following

• A TR can be configured such, that userland application receive on-time information from the timing system ("timing event").
• Configure System-on-Chip components to execute on-time actions such as digital output.
• Provide access to configurable I/O - direction (in/out), termination (on/off).
• Provide means for generating pulses, clocks, delays - triggered by combination of digital input or "timing events".
• Provide statistics such as # of processed timing messages, # of errors, ...

USB support

Since version 2.2.4 saftlib can work with Timing receivers attached via USB.

TL;DR: When using saftlib with a USB timing receiver (e.g. saftd tr0:dev/ttyUSB0) all eb-tools must use $(saft-eb-fwd tr0) instead of dev/ttyUSB0 (e.g. eb-ls $(saft-eb-fwd tr0)).

A timing receiver attached via USB is controlled through a generic serial driver in Linux which does not support multiple etherbone connections at the same time. When saftlib is running, the saftd process always occupies the connection. If another etherbone tools uses the same device, both, the tool and saftd will crash. Therfore, saftd creates a pseudoterminal (e.g. /dev/pts/8), that emulates an etherbone device and forwards all data to the real hardware device. This is done in a way that all etherbone transactions are completed and the communication will work for all eb-tools using this pseudo terminal. In Linux, the pseudoterminal devices are named /dev/pts/x, where x is an integer number. There is a saftlib tool (saft-eb-fwd) to find out the name of the pseudoterminal for a given saftlib device (e.g. tr0). This can be used to replace the device name (e.g. dev/ttyUSB0) when calling an eb-tool (e.g. eb-ls $(saft-eb-fwd tr0)).

Plugins

Since version 3.0.0 saflib is chaned to support plugins. The inter process communication is provided by a daemon (saftbusd) and all services must be dynamically loaded into this daemon using plugins (shared object libraries). The saftlib services in previous versions were baked into the saftd executable. Since version 3.0.0 the same functionality is obtained by starting saftbusd executable and loading the libsaft-service.so plugin. The saftlib repository also contains plugins for the function generator and burst generator services.

Environment Variables

Since version 3.0.0 the daemon (now saftbusd) uses a deterministic memory allocator. This can be avoided by using a different executable (saftbusd-noda). If the deterministic memory allocator is used, it can be configured with the environment variable SAFTD_ALLOCATOR_CONFIG. The default configuration is "16384.128 1024.1024 64.16384" (pairs of integers, connected with a point, separated by whitespaces) which means

• 16384 blocks of size 128 bytes
• 1024 blocks of size 1024 bytes
• 64 blocks of size 16384 bytes

Allocations larger than the larges block size fall back to the default heap allocator. Number of blocks must strictly decrease. Block size must strictly increase.

Since version 2.4.0 the FunctionGenerator devices use boost::circular_buffer instead of std::deque as underlying container for the data-FIFO. The initial capacity of the circular_buffer can be controlled with the environment variable SAFTLIB_FG_FIFO_MAX_SIZE.

• default capacity of all FIFOs is 16384
• the capacity will be increased automatically during runtime whenever it is too small. Ideally the initial size is large enough such that this will never happen.

Since version 2.2.0 the location of saflib's IPC socket (saftbus) can be changed with the environment variable SAFTBUS_SOCKET_PATH.

• default location for the saftbus socket is /var/run/saftbus/saftbus
• can be changed by setting e.g. export SAFTBUS_SOCKET_PATH=/tmp/saftbus
• prior to verions 2.2.0 the saftbus socket location was /tmp/saftbus and could not be changed by users

Since version 2.1.0 a file with a leap-seconds list ist needed

• This is nomally installed to /share/saftlib/leap-seconds.list
• A different file can be used by setting the environment variable SAFTLIB_LEAPSECONDSLIST , e.g. export SAFTLIB_LEAPSECONDSLIST=/tmp/my-leap-seconds.list

Concept and more Details

Figure: SaftLib with (left) and without (right) D-Bus. Userland software is shown in blue and FPGA based hardware is shown in green.

A simiplified concept of SaftLib is shown in the figure above. Features of SaftLib are implemented using a saftd daemon that has direct access to the hardware. The userland applications saft-ctl and a FESA class communicate with the hardware via saftd. Originally, the required inter-process communication has been implemented on D-Bus (more information on D-Bus withing SaftLib can be found here).

As indicated on the right side of the above figure, D-Bus is being replaced by pipes and sockets (autumn 2018). This has the following advantages.

• reduced latency
• simplified real-time scheduling of userland processes
• simpler deployment as run-time dependencies get reduced

Starting spring 2019, SaftLib runs without D-Bus and time (uint_64 t t //nanoseconds since 1970) is replaced by a dedicated 'time' datatype. Please check-out the migration guide from version 1.4 -> 2.0 (see references below).

Diagnostic Logging

Since version 2.4.0 saftd does diagnostic logging in a fixed-size ring-buffer. This buffer is dumped into graylog whenever an exception is thrown by one of the device drivers. In order to limit the number of messages towards graylog, such a dump can happen only 10 times. After that, the number of dumps has to be reset manually. The logdump can be redirected into a local file. All settings related to this logging can be controlled with the tool "saftbus-ctl". Type saftbus-ctl --help for details.

Remote Debugging with GDB

A HOW-TO can be found here HOW-TO: connect GDB to saft-daemon on SCU

Documentation

Documentation on the software API is autogenerated by Doxygen when building SaftLib from the master branch (see notes on how-to build saftlib building here). For convenience, a pdf version of the documentation can be generated by typing the following three commands in the main saftlib folder

• doxygen
• make -C latex
• evince latex/refman.pdf

The Doxygen documentation is available here and pdf version of the documentation is available in ref [3] below.

There exist a couple of useful command line tools based on SaftLib. An overview is given in this How-To.

References

[1] F-DS-C-06e, Detailed Specification of the FAIR Timing Receiver Nodes

[2] Migration Guide 1.4 -> 2.0; presentation, wiki

[3] documentation, pdf-v3.0.1 (previous versions: pdf-v2.4.3 (2021), pdf-v2.2.4 (2020), outdated (2019))

-- DietrichBeck, MichaelReese - 28 August 2023