How-To: Building and Deployment for Linux Boxes and SCU

Most of the code has been developed in the context of the White Rabbit Project and is hosted by the Open Hardware Repository. For integration into the environment specific to GSI, a dedicated repository has been created on GitHub. This repository includes the codes from the OWHR site by using submodules.

Intended usage: F-Release - For older releases please check-out the history of this Wiki page.

This how-to describes building and deployment of the timing system for various platforms.

This how-to does not describe setting up a Linux box. This is covered by a dedicated how-to.

Introduction

The concept of the FPGA gateware of timing receiver nodes is based on using a Wishbone bus architecture that allows accessing the functionality of the various IP cores instantiated on the FPGA. The internal system on-chip Wishbone bus is extended to the outside world using Etherbone. Thus, accessing the resources of the timing receiver from a user-land application requires a host-bus to Wishbone bridge. At present, the approach looks like this

[---------------------------------------FEC-------------------------------------]
[-----TR-----]...[--------------------------host--------------------------------]
[----FPGA----]...[-------kernelspace-------]...[-----------userspace------------]
[SoC<->bridge]<->[kernel_mod<->wishbone_mod]<->[libetherbone<->saftlib<->program]  # use this!
                                                             |
                                                             ->program]            # not recommended

Figure: A Front-End Computer (FEC) typically includes a Timing Receiver (TR) with FPGA as central component and a host. The host system is typically a PC (Com-Express, VME CPU, ...) and runs a Linux operating system. The driver for a TR uses a wishbone module and a host bus specific kernel module. User programs, saftlib and Etherbone library are userland applications. The host-bus bridge can be any sort of serial protocol, like Ethernet, USB, PCIe ...

Development

Repository (bel_projects)

The URL of our project on GitHub is

https://github.com/GSI-CS-CO/bel_projects.git

Fresh Checkout of the Repo For a fresh checkout of the repository do

git clone < URL >

Navigate to Desired Branch In case you don't like to work on "master" but on another specific branch do the following. Warning: This might overwrite local changes: In case you have done local changes consider to "stash" them before. Info: Try git remote show origin to check for existing branches.

cd MYPATH/bel_projects
git checkout < BRANCH >
make

The most important branches are

• master - the "master" branch
• proposed_master - the main branch for development
• branch for a specific release, see here
• branch for the current release, see here

Update from the Repo For updating the local project from GitHub do

cd MYPATH/bel_projects
git stash
git pull
make

Prepare for Quartus The following is only required for building bitstreams from VHDL code.

$ export HDLMAKE_QUARTUS_PATH=/path/to/quartus/bin
$ export PATH=$PATH:HDLMAKE_QUARTUS_PATH

And be sure that the Quartus license file is exported too:

$ export LM_LICENSE_FILE=/path/to/License.dat

Trouble? In case "make" throws errors at you which you can't fix, try one or more of the the following options.

• Try to update from your local repository (git checkout BRANCH).
• Reset your git workspace git reset --hard . This discards any modifications done to files under version control. Any changes to tracked files in the working tree since the last commit are lost.
• Clean up your git workspace git clean -xfd. This cleans the working tree by recursively removing all files and folders that are not under version control.
• Another trick might by to make distclean . Warning: "distclean" does git clean -xfd (see above) and also removes untracked files.
• If nothing helps: Consult the ultimate git manual (delete your project and download a fresh copy).

Figure: GIT cheat sheet (repo, pdf).

Build and Installation

The main idea is to use the "Makefile" in MYPATH/bel_projects. Have a look at that makefile to see all options. The followings just depicts a few use-cases.

NB: The GIT repo makes use of sub-projects. Some of these (saftlib, Etherbone) have already been migrated now using the GNU "autogen" and "configure" tools.

Uninstall

There is no dedicated uninstall. Prior a 'fresh' installation, do double-check you cleaned away any remains of previous installs. Carefully go through all locations (or search your filesystem). Usually, stuff gets installed like this.

• PREFIX/bin
• PREFIX/include
• PREFIX/lib
• PREFIX/lib/pkgconfig
• PREFIX/sbin (some older versions of saftlib)
• PREFIX/share

PREFIX is usually defined at build time. However, sometimes one forgets to define PREFIX. And there have been reports, that stuff gets installed in default paths instead of the path defined by PREFIX. Double check the usual locations for PREFIX, such as

• /usr/local
• /opt/usr
• ?

Usually, binaries and libraries can easily be recognized by their prefixes

• eb-
• saft
• saft-
• libether
• libsaft
• etherbone

With newer saftlibs there are additional files to be deleted

• soft-tr
• software-tr
• wait-msi

and other libraries to be deleted, which can be identified by their prefix

• libbg
• libfg
• bg

and other includes to be deleted (some of them are folders)

• bg
• bg-firmware
• fg-firmware
• saftbus
• saftlib

Standalone PC

This assumes you have sudo privileges on a local PC.

Ideal Situation If things work best, just do the following.

cd MYPATH/bel_projects
make clean
make
sudo make install
make saftlib-clean          //optional
make saftlib                //optional
sudo make saftlib-install   //optional

In case "make" does not throw any error message at you, you are fine. Now you can connect/install your hardware and you are done.

Minimal Installation In case you are only up for a minimal installation (or "make all" throws errors at you) just try to install Etherbone, our tools and the Linux drivers.

cd MYPATH/bel_projects
make clean
make etherbone
make tools
make driver
sudo make install
make saftlib-clean          //optional
make saftlib                //optional
sudo make saftlib-install   //optional

Linux Boxes with Installation in "/opt" (such as boxes provided by central IT of GSI)

For other purposes like non-standard install paths, cross-compiling or missing sudo privileges, the following environment variables are useful for building, installing or creating tar-balls.

• PKG_CONFIG_PATH : We use pkg-config for metadata on installed libraries of our local system. As we don't have sudo priviliges, we need to maintain the metadata in a non-standard path specified by the PKG_CONFIG_PATH environment variable.
• KERNELDIR : Path to kernel headers/sources. Only required for building drivers
• PREFIX : This controls, where compiled programs and libraries do expect stuff they depend on. Use this environment variable to avoid later usage of LD_LIBRARY_PATH.
• STAGING: Defines, where make will install the programs and libraries after they have been build. This is useful for installing to non-standard paths or building tar files as distribution.

Here it is assumed, one has sudo priviliges, and all user installation should be done in "/opt". For our installation we choose "/opt/usr" as base path.

Set environment variables:

export PKG_CONFIG_PATH=/opt/usr/lib/pkgconfig
export PREFIX=/opt/usr

Build and Install Etherbone, Tools, Drivers

cd [FOLDER OF BEL_PROJECTS WITHIN GIT CHECKOUT]
make clean
make PREFIX=/opt/usr               // hm..., it seems PREFIX is required as 'eb tools' might get installed in the default path instead
sudo make PREFIX=/opt/usr install  // 'sudo' is required for drivers; but also installs 'eb tools' etc to /opt/usr as root

Build and Install SaftLib

cd [FOLDER OF BEL_PROJECTS WITHIN GIT CHECKOUT]
make saftlib-clean
make saftlib EXTRA_FLAGS=--prefix=/opt/usr    // consider using a trailing '-j4' for faster build
make saftlib-install EXTRA_FLAGS=--prefix=/opt/usr 

Start saftLib daemon as sudo.

sudo /opt/usr/sbin/saftd BOARDNAME:dev/wbm0 >/tmp/saftd.log 2>&1

RPI (not yet released)

This works similar to standalone PC.

• Ubuntu Server 20.04.2 LTS
• follow directions given in README.md in the git repo https://github.com/GSI-CS-CO/bel_projects/blob/master/README.md
• you need additional packages
  • build-essential
  • autotools-dev
  • autoconf
  • automake
  • libtool
  • docbook-utils
  • readline-dev
  • libboost-dev
  • sigc++
  • xsltproc

In principle this works, but this is neither released nor tested.

Some hints:

• make clean
• make (this will throw a few errors at you (lm32-toolchain) which should be ignored for now)
• make etherbone
• make tools
• make driver (this will throw an error about PCIe drivers at you)
• edit ip_cores/fpga-config-space/pcie-wb/Makefile

obj-m += pcie_wb.o wishbone.o            // replace this line by ...
obj-m +=  wishbone-serial.o wishbone.o   // this one

• make driver
• sudo make install
• sudo make saftlib-clean
• make saftlib (this will take some time)
• sudo make saftlib-install

Building Test-Binaries for SCU

Rocky9 / Yocto

Binaries shall be built using the Yocto SDK

SL7

Disclaimer
Though shall build stuff using ci_cd, in case your stuff is intended for production.

The recipe described in the following may only be used to build binaries or shared libraries for testing or quick hacks (not for production).

Idea
The idea is to build and install into a 'fake' /usr directory in your home folder on ASL. You can then use scp to copy binaries or libraries from that folder to the target system.

Note
We don't use the 'STAGING' method, as this messes up the Etherbone package config in PKG_CONFIG_PATH.

Here it is assumed

• you have an ASL account without sudo privileges and your are logged on
• /home/bel/USER/lnx/bel_projects is the location of your 'bel_projects' branch
• /home/bel/USER/lnx/staging is your fake installation directory

Set environment variables:

export PREFIX=/home/bel/USER/lnx/staging/usr
export PKG_CONFIG_PATH=/home/bel/USER/lnx/staging/usr/lib/pkgconfig

Build and perform fake installation:

cd bel_projects
make
make install   // this will throw a few erros about missing privileges, those can be ignored
make saftlib
make saftlib-install

Your stuff is now available in the folder defined by PREFIX.

Runtime Deployment for SCU on ASL (using ci_cd)

An outdated HOW-TO for manual deployment is still available here.

Prerequisites

The recipe describes how-to provide software and drivers for the SCU runtime environment starting from the control system release 8 (autumn 2016). There is no dedicated cross-compiler for the SCU. Instead, all compilation takes place on the ASL cluster that runs the same Linux version. Logon to a machine on the ASL cluster before proceeding, EL7: asl74x.

Luckily, preparing the runtime environment for SCUs is basically done via scripts that are provided within a GIT repo.

Repository (ci_cd)

The URL of the project on GitHub is

https://github.com/GSI-CS-CO/ci_cd.git

Fresh Checkout of the Repo For fresh checkout of the repository do

git clone --recursive URL

Navigate to Desired Branch ... if required. For directions on how to change to a branch, see above.

Update from the Repo For directon on how to update the repo, see above.

Build and Preparation of Deployment

The main idea is to use the script build-rte.sh in MYPATH/ci_cd/scripts/deployment/RTE_Timing .

At present, the script does not support any features like auto-detection the installation folders and so on. Edit the script accordingly. Most importantly, you want to modify the scripts including things such as

• BEL_BRANCH="enigma" (modify to get a different release)
• BEL_RELEASE="" (optional; "" uses 'head'; you may also use a 'tag' or 'sha')
• DEPLOY_TARGET="/dev/null" (modify to your needs; example: /common/export/timing-rte/tg-enigma-v5.0.2-beta

In case the control system migrates to a new version of the operating system, you also need to modify the kernel info

• LINUX_KERNEL="linux-scu-source-3.10.101-03"
• KERNEL="linux-scu-source_3.10.101-03"

Installation Just start the script, et voila:

[user@asl4711 RTE_Timing]$ ./build-rte.sh

Check the result

• The command line output of the script is written to a log file RTE_log_DD-MM-YYYY_hh-mm-ss.
  • Check for build errors via cat RTE_log_DD-MM-YYYY_hh-mm-ss | grep error.
  • Check the final messages tail RTE_log_DD-MM-YYYY_hh-mm-ss
• Check the contents of the file /common/export/timing-rte/tg-enigma-v5.0.2-beta/x86_64/etc/timing-rte_buildinfo
• special : do some hacking such as https://github.com/GSI-CS-CO/bel_projects/issues/167

Create loader script for nfs-init

• simply copy an existing script and name it accordingly
• modify the content of the script to point to the correct folder in 'common/export/timing-rte/...'

Protect (Optional) To exclude erroneous modification of the runtime system by others, remove write access from all other group members.

cd /common/export/timing-rte
chmod g-w -R R8-balloon_0

Backup To be on the safe side backup the runtime system. Moreover, backup some RPMs that were pulled and used by the script build-rte.sh.

cd /common/export/timing-rte
tar -cf R8-balloon-0_2016-dec-16.tar R8-balloon_0/

cd MYPATH/ci_cd/scripts/deployment/RTE_Timing/rte-tmp/
tar -cf rte-tmp_lib-rpm_kernel_2016-dec-16.tar lib/ rpm/ linux-scu-source-3.10.101-01/

Move that tar files to a safe place.

Deploy

Simply set some symbolic links for the FEC and reboot, see here.

Example: Verify things work

etherbone and driver

$ eb-mon -v dev/wbm0
EB version / EB source: etherbone 2.1.0 (v2.1.0-4-g809617b): Apr 29 2016 02:26:04 / built by dbeck on Nov 28 2016 13:12:37 with asl742.acc.gsi.de running CentOS Linux release 7.2.1511 (Core) 
WR_time - host_time [ms]: -1476162305285
Current TAI:1970-02-18 10:50:31 GMT
Sync Status: TRACKING
MAC: 00267b0003b2
Link Status: LINK_UP
IP: 192.168.16.175

In case the above does not work, try eb-ls dev/wbm0 first.

saftlib

$ saft-ctl blabla -fijs
saftlib source version                  : saftlib 1.0.8 (v1.0.8-40-g4734cda): Oct 27 2016 10:20:55
saftlib build info                      : built by dbeck on Nov 28 2016 13:13:36 with asl742.acc.gsi.de running CentOS Linux release 7.2.1511 (Core) 
devices attached on this host   : 1
  device: /de/gsi/saftlib/baseboard, name: baseboard, path: dev/wbm0
  --gateware version info:
  ---- Tue Nov 22 02:11:28 CET 2016
  ---- developer preview
  ---- Arria II GX (EP2AGX125EF29C5)
  ---- Debian GNU/Linux 8.6 (jessie), kernel 3.16.0-4-amd64
  ---- scu3 +comexpress
  ---- Jenkins Nightly Build <csco-tg@gsi.de>
  ---- tsl002.acc.gsi.de
  ---- scu_control
  ---- Version 16.0.0 Build 211 04/27/2016 SJ Standard Edition
  ---- zenith-1262

WR locked, time: 0x000edf752496f5e0
receiver free conditions: 256, max (capacity of HW): 0(256), early threshold: 4294967296 ns, latency: 4096 ns

Some Hints

VME

We no longer provide timing receivers in VME form factor. The VME driver is kept alive only with 'best effort'. In case you want to use one of the old VETERA2a boards that are still around, you have to cross compile the driver for the architecture and install it in the kernel tree that you are going to use for booting the host system. All the instructions above applies also for the building and installation of the VME driver, you may have problems building the driver if you have a target kernel > 3.10.

Modifying Makefiles

Starting with saftlib, many Makefiles are autogenerated using the autogen tools. Changes must not be performed in the Makefiles, but from the "sources" used by autogen. As an example, for adding a .c file to a build target, do so in "Makefile.am".

-- DietrichBeck - 15 February 2023