How-To: White Rabbit UNILAC PZ (wr-unipz) - Coding and Deployment

Introduction

"wr-unipz" is a component of the MIL-based UNILAC 'Pulszentrale' (UNIPZ). As a field-bus, it does not use the MIL 'Event' bus but a White Rabbit network. Logically, wr-unipz is identical and on the same level as the seven 'Pulszentralen' (PZ) of the UNILAC timing system. As for the other PZ, data supply and 50 Hz synchronization are provided through the 'Super Pulszentrale' (SPZ) of the UNILAC timing system. For technical and conceptual reasons, the UNILAC timing system and the GSI/FAIR General Machine Timing system (GMT) must use two distinct White Rabbit networks.

How-To

• WR-UNIPZ
  • operation - useful for collegues from operation, includes a list of error codes
  • coding, building and deployment - useful for developers
  • off-topic: some hints for Device Access - useful for newbies on the old control system, required info for modifying the Device Access part
• MIL-UNIPZ
  • some practical hints for diagnostics - explains also nomen for UNIPZ
• Users at UNILAC
  • identical interface as for the FAIR timing system
    • same timing receivers (SCU, Pexaria...)
    • same drivers, Etherbone tools, saftlib
  • differences
    • timing messages have a 'UNILAC dialect', see here
    • there are just UNILAC events and no such thing as CMD_SEQ_START ...

Documentation

• WR-UNIPZ
  • documentation
• MIL-UNIPZ
  • documentation by Peter Kainberger und Udo Krause (a bit out of date)
  • 'MIL timing': presentation by Udo Krause (consider this as an update to the documentation of UNIPZ)
  • how-to by DB

Overview

Figure: Integration of White Rabbit Pulszentrale into UNILAC Pulszentrale. Shown are the existing components Settings Management (grey box), Super Pulszentrale (light green box), Pulszentralen 1..7 (green boxes) and the new White Rabbit Pulszentrale (blue box). As well shown are MIL event bus lines (green arrows), a White Rabbit network (blue arrows) as well as connections from Super Pulszentrale to PZs (black arrows). Details see text.

Today (2020) the UNILAC is still operated using the 'old GSI control system' Device Access . The timing system is based on a so-called MIL Event Bus with a Pulszentrale (PZ) as master. UNILAC has seven timing areas with one PZ each. Thus, there are seven distinct MIL cables, one for each PZ. The seven PZs are coordinated via the Super Pulszentrale (SPZ), which has two tasks.

• It supplies the PZs with data. The data contain information for up to 16 independent virtual accelerators (VACC). For each VACC there are two distinct sets of data (called 'Kanäle'), one for normal operation and one for low intensity operation ('verkürzt', 'Profilgitterschutz'). In total the SPZ needs to provide 7 * 16 * 2 = 224 event sequences. Each sequence is a list of event data and a time offset relative to the beginning of a UNILAC cycle. SPZ transmits the data to the PZs using Device Access via the standard ACC network.
• SPZ and the PZs are interconnected via an internal MIL event bus. SPZ uses this internal bus for various purposes:
  • announce event: announces the virtual accelerator number that must be played for a specific PZ during the next cycle; there may be up to seven of these events per cycle (one for each PZ)
  • cycle start event: starts the next UNILAC cycle. Each cycle has a length of ~20ms (50 Hz); there is one event per cycle starting the event sequences at (up to) seven PZs simultaneously
  • service event: service events are played after the event sequence (at a PZ) has been completed, there is one event for each service event to be sent
  • synch data event: if new data have been supplied via Device Access, they will become active during the next cycle; this event is processed by all PZs

The White Rabbit PZ just behaves like the seven PZ. It is supplied via Device Access and is connected to the internal MIL Event Bus to receive the events from SPZ. There are two main difference to the seven PZs.

• there is only one White Rabbit network for all seven timing areas at UNILAC
• White Rabbit PZ uses the concept of an alarm based timing system unlike the seven PZs, that are event based

Stack of the White Rabbit Pulszentrale

Figure: Overview on the White Rabbit Pulszentrale. Shown are SCU (blue box), host system and FPGA (orange boxes), standard IP cores instantiated in the FPGA (light green boxes), components by the WR-UNIPZ (yellow boxes) and components of the old GSI Control System (dark green boxes). Indicated is data flow from the Super Pulszentrale (thick black arrows), a Lemo cable (thin blue arrow), Wishbone and Etherbone communication (thin black arrows) and Timing Messages sent to the White Rabbit network (thick blue arrow). Details see text.

One of the main challenges of this development was to bring together two different worlds: The old GSI Device Access control system and the very different ecosystem of the White Rabbit based developments. The low level interface to timing system components is Etherbone. The high level standard API is saftlib. However, WR UNIPZ uses a third approach, namely the use of a shared library. This disentangles the two distinct ecosystem as much as possible:

• Developing, building of the firmware and tools works independently of the old GSI control system.
• Developing, building of the Device Access class only requires including one header file and linking against a single shared library.

Regular SCU
The basis of WR-UNIPZ is a regular SCU as Frontend with Timing Receiver. Hardware, gateware and the RunTime Environment (RTE: drivers + tools) are identical to other frontends. This is configured and rolled-out like all other SCU based frontends.

WR-UNIPZ: Firmware and Tools
Very tightly coupled to the gateware is firmware and low-level software tools of WR UNIPZ that are hosted in the same source code control and build environment as gateware and RTE. It is part of the White Rabbit based ecosystem. The tools contain all a script and helper software to upload the firmware to a lm32 soft CPU embedded in the VHDL of the regular SCU during runtime. Moreover, the script performs basic configuration of firmware, IOs ports and the ECA. Configuration and roll-out works via nfs-init (as usual) and includes a shared library wrunipzlib.so as API. This shared library hides implementation details as much as possible from the user. Included as well is a CLI for testing and diagnosis.

Precise timestamping needs to be done for the 'cycle start event' received from the SPZ via MIL bus. This is achieved by the Timestamp Latch Unit (TLU) of the ECA. The 'MIL Piggy' generates a TTL pulse on an output when the cycle start event is received. Via a LEMO cable, this TTL signal is connected to an input configured as TLU input of the ECA. This solution provides timestamping in hard real time by the gateware, indepently of the firmware.

PZUD: Device Access Class
The equipment class pzud is part of the Device Access environment and part of the equipment model pzu. pzud only uses the shared library wrunipzlib.so as the sole interface to the lm32 firmware in the FPGA. Basically, pzud can be seen as a wrapper around that shared library.

WR-UNIPZ

NFS-Init

Configuration of the frontend is done via nfs-init, here for the SL7 ramdisk

Example for PRO:

[me@asl789 scuxl4711]$ ll
total 4
lrwxrwxrwx 1 me      bel  49 Sep  5  2023 10_timing-rte -> ../global/timing-rte-tg-fallout-v6.2.0-no-graylog      // timing RTE
lrwxrwxrwx 1 me      bel  29 May 25  2023 20_wrunipz_wrunipz-tools -> ../global/timing-rte-db-yocto               // tools
lrwxrwxrwx 1 me      bel  29 May 25  2023 30_wrunipz_wrunipz-pro-config -> ../global/timing-rte-db-yocto          // launch and config firmware
lrwxrwxrwx 1 me      bel  29 May 25  2023 40_wrunipz_wrunipz-pro-systemd -> ../global/timing-rte-db-yocto         // launch monitoring server
lrwxrwxrwx 1 person  bel  16 Sep 21  2020 90_devacc -> ../global/devacc                                           // launch device access software

Example for INT, note the differences in the names of the symbolic links (click):

[me@asl789 scuxl4712]$ ll
total 4
lrwxrwxrwx 1 me      bel  49 Sep  3 15:56 10_timing-rte -> ../global/timing-rte-tg-fallout-v6.2.0-no-graylog        // timing RTE
lrwxrwxrwx 1 me      bel  29 May 25  2023 20_wrunipz-dev_wrunipz-tools -> ../global/timing-rte-db-yocto             // tools
lrwxrwxrwx 1 me      bel  29 May 25  2023 30_wrunipz-dev_wrunipz-int-config -> ../global/timing-rte-db-yocto        // launch and config firmware
lrwxrwxrwx 1 me      bel  29 May 25  2023 40_wrunipz-dev_wrunipz-int-systemd -> ../global/timing-rte-db-yocto       // launch device access software
                                                    ^^^^         ^^^ 
                                                       |           |- environment
                                                       |- path 
lrwxrwxrwx 1 person bel  16 Sep 11 09:01 90_devacc -> ../global/devacc                                              // launch device access software

Code, Build, Deployment

Code is here: bel_projects/tree/.../modules/wr-unipz

For branch and hash of the current version check the logbook

For building and deployment check the comments given at the top of the 'Makefile'

Example for building and deployment for PRO (assuming target system runs Yocto ramdisk)

make clean                                                                  // clean
make ENV=pro SYSENV=ACC7 PREFIX= all                                        // builds software, firmware and generate nfs-init scripts
make SYSENV=ACC7 PREFIX= STAGING=/common/export/timing-rte/wrunipz deploy   // copies stuff to location where nfs-init will find it

PZUD

This information is deprecated as ASL6 is no longer available.

In case you are not familiar with Device Access and the build environment on the ASL >>> 6 <<< cluster, checkout this.

Standard Device Properties

Some info on coding...

pzud-device.cc provides some data and standard device properties

• data
  • ebDevice, handle to Etherbone device of pzud
  • pz, number of PZ (0..6); there is one instance of the PZUD class for each PZ
• methods
  • resetDevice, init of all get-values; here: reset diagnostic values (for all PZ)
  • initDevice, as reset (for all PZ), but init of all set-values too; here: clears event data (for THIS PZ)
  • updateStatus, is triggered periodically to get and update the device status; here: get basic info (for ALL PZ, 'opReady'....)
  • onlineTest, is triggered periodically to get and update the device status; here: get very basic info (for ALL PZ, check if firmware is reachable)
  • getVersionString, obtains version info; here: PZUD, version, shared library

USRs (User Service Routines) provide methods dedicated to PZUD only; USRs are defined in the file pzud.xml

• pzud-write-evtsequ.cc, writes event data for a specific virtual accelerator and channel to the firmware, important method used by the 'UNILAC Super Pulszentrale'
• pzud-read-evtsequ.cc, reads event data for a specific virtual accelerator and channel to the firmware
• pzud-read-diag.cc, reads diagnostic info of firmware (for ALL PZ)
• pzud-write-cmdcleardiag.cc, clears diagnostic info of firmware (for ALL PZ)
• pzud-write-cmdsubmit.cc, submits/activats event data previously written to the PZs (for ALL PZ); remark: this method is only useful when playing with PZUD via the PropHelper. The 'Super Pulszentrale' uses a dedicated event on the internal MIL bus to which PZUD is connected.

Interface to the lm32 Firmware in the FPGA

There is only very little code implemented by PZUD. Basically, PZUD is just a wrapper around a shared library 'wrunipzlib.so'. This library provides the interface to the firmware including all communication. The library and headere file are provided here

• /common/export/timing-rte/wrunipz-dev ; development system BG2.009, H2O Racks
• /common/export/timing-rte/wrunipz; production system LSB6

The Device Access class PZUD just includes the header and links against the shared library (see the info on 'SConscript' here).

Building and Roll-Out

Follow these directions.

-- DietrichBeck - 13 Nov 2024