How-To: Flash a Timing Receiver with a Gateware/Firmware Image

TL;DR

• disable all software on the host (FESA, saftd, ...)
• SCU only: disable the watchdog using eb-reset < device > wddisable
• double check you have the correct rpd file!!!
• eb-flash-secure < device > < rpdfile >
• if things went fine: eb-reset < device > fpgareset

Introduction

This how-to is intended for non-experts and describes how to flash a timing receiver with a new image. There is also a presentation on this topic.

Fig 1: Flash a Timing Receiver.

An overview on the procedure is depicted in Fig. 1. and basically includes four steps

1. Inspect what image is actually loaded into the FPGA.
2. Upload a rpd file containing the new image to the host system of the FEC.
3. Write the image to the flash chip.
4. Power-cycle the Timing Receiver to load the new image to the FPGA.

Inspect and Upload

For preparation one needs to identify, download and copy the file of the new image to the file system of the FEC.

• Please identify what kind of form-factor should be flashed (SCU2, SCU3, PEXARIA, ...). Do this via a unique identifier (1-wire ID, visual inspection...)
• Gain access to the host system via ssh login. Verify what image is active in the Timing Receiver using eb-info.

[ruth@scuxl0815 ~]# eb-info dev/wbm0
Project     : scu_control
Platform    : scu3 +comexpress
FPGA model  : Arria II GX (EP2AGX125EF29C5)
Source info : balloon-1288
Build type  : developer preview
Build date  : Fri Nov 18 04:48:42 CET 2016
Prepared by : Jenkins Nightly Build <csco-tg@gsi.de>
Prepared on : tsl002.acc.gsi.de
OS version  : Debian GNU/Linux 8.6 (jessie), kernel 3.16.0-4-amd64
Quartus     : Version 16.0.0 Build 211 04/27/2016 SJ Standard Edition

  00f7a65 Merge branch 'audio_pll_upgrade' into balloon
  51cef37 nau8811: updated core and tools
  1deb67c nau8811: updated ip core (quartus 16)
  1885c91 Merge remote-tracking branch 'origin/nau8811_20160713' into audio_pll_upgrade
  6231918 flow/fix-git: tired of git submodule issues

• Closely inspect the information listed in Project and Platform. In this example, the form factor is a SCU3. Consequently, a rpd file named scu_control3[additional identifier possible].rpd will be needed.
• Download the rpd file following the instructions of the current release, or choose from all releases. Adventurous people might want to try a nightly build from the continuous integration system.
• Copy the rpd file to the FEC using scp

[me@asl742 ~]$ scp scu_control3.rpd ruth@scuxl0815:.
ruth@scuxl0815's password: 
scu_control3.rpd                                        100%   16MB  16.0MB/s   00:01    

• Having logged on to the FEC via ssh, verify the image has been successfully transferred.

[ruth@scuxl0815 ~]# ls -l
total 16384
drwxr-xr-x    2 root     root             0 Aug 11 10:05 bin
drwxr-xr-x    6 root     root          2380 Nov 22 17:07 dev
drwxr-xr-x   14 root     root             0 Nov 22 17:07 etc
lrwxrwxrwx    1 root     root            11 Aug 11 09:46 init -> bin/busybox
drwxr-xr-x   13 root     root             0 Nov 22 17:07 lib
lrwxrwxrwx    1 root     root             3 Aug 11 09:46 lib64 -> lib
drwxr-xr-x    5 root     root             0 Nov 22 17:07 opt
dr-xr-xr-x  100 root     root             0 Nov 22 17:07 proc
drwx------    2 root     root             0 Aug 11 10:02 root
drwxr-xr-x    2 root     root             0 Nov 22 17:07 sbin
-rw-r--r--    1 root     root      16777216 Nov 23 15:47 scu_control3.rpd
dr-xr-xr-x   12 root     root             0 Nov 22 17:07 sys
drwxrwxrwt    2 root     root             0 Nov 22 17:07 tmp
drwxr-xr-x    6 root     root             0 May 30 09:30 usr
drwxr-xr-x    9 root     root             0 Nov 22 17:07 var

Stop all Firmware and Software

It is safer, to stop all firmware and software gently. Then, operations on the flash won't be in competition with concurrent stuff in the same FPGA.

• FESA
• other application software
• socat
• saftlib
  • saft-ctl tr0 quit
• lm32 firmware
  • firmware by DB: use xxx-ctl dev/wbm0 stopop followed by xxx-ctl dev/wbm0 idle to get firmware into a safe IDLE state
  • firmware by other devlopers: ask
  • invoke eb-reset to halt all lm32 cores (i.e., eb-reset dev/wbm0 cpuhalt 0xff)

Disable the FPGA Watchdog on SCUs

Starting with timing release 6.2.0, a new watchdog feature has been deployed which will reset the FPGA in 10 minutes in case the host system or saftd is dead. To prevent an accidental reset during flashing, we should disable the watchdog on the SCU prior flashing

[root@scuxl0815 ~]# eb-reset dev/wbm0 wdstatus   // probe watchdog
1                                                // result '1': watchdog is enabled
[root@scuxl0815 ~]# eb-reset dev/wbm0 wddisable  // disable watchdog
[root@scuxl0815 ~]# eb-reset dev/wbm0 wdstatus   // probe watchdog
0                                                // result '0': watchdog is disabled

Write the Image to the Flash

• Having logged on to the FEC via ssh, once again use eb-info and ls -l to verify the rpd file name matches the actual image of the FPGA.
• Look at the parameters expected by the tool eb-flash

[ruth@scuxl0815 ~]# eb-flash 
eb-flash: expecting three non-optional arguments: <proto/host/port> <firmware>
[ruth@scuxl0815 ~]#

• instead of using the native tool eb-flash, it is recommended to use the script eb-flash-secure; eb-flash-secure is available with timing release enigma or later

[ruth@scuxl0815 ~]# eb-flash-secure dev/wbm0 scu_control3.rpd 
Autodetecting sector size: found = 0x40000         
Scanning offset 0x21ffff4c ... done!
Erasing 0x21fc0000 ... done!
Programming 0x21ff0000... done!
Verifying 0x21ff0000... done!

Don't power cycle or reset the FPGA now!

Load the Image into the FPGA

Review the Situation

Now, the FPGA up still using the previous image. The new image has only been written to the flash chip but is not yet loaded into the FPGA.

This is your last chance to replace the image in the flash chip by another one or to fix errors that might have occurred.

In case of real trouble, this is the time to call the experts. They might be able to fix a problem, while the FPGA is still active with the old image.

In case problems showed up until this step, don't power cycle or reset the FPGA. Otherwise you might end up with a bricked device.

Final Steps

• Load a new image to the FPGA by
  • power-cycling your FEC including the host system. (Remark: A simple reboot of the host system alone will just shutdown and fire up again your OS but has no effect on the FPGA.)
  • SCU: 'eb-reset dev/wbm0 fpgareset'
  • SuperMicro
    • clean: power cycle via management interface
    • dirty: 'eb-reset dev/wbm0 fpgareset; reboot'
  • ...
• Logon to the FEC via ssh and verify the new image has been loaded from the flash chip to the FPGA using eb-info

[ruth@scuxl0815 ~]# eb-info dev/wbm0
Project     : scu_control
Platform    : scu3 +comexpress
FPGA model  : Arria II GX (EP2AGX125EF29C5)
Source info : zenith-1262
Build type  : developer preview
Build date  : Tue Nov 22 02:11:28 CET 2016
Prepared by : Jenkins Nightly Build <csco-tg@gsi.de>
Prepared on : tsl002.acc.gsi.de
OS version  : Debian GNU/Linux 8.6 (jessie), kernel 3.16.0-4-amd64
Quartus     : Version 16.0.0 Build 211 04/27/2016 SJ Standard Edition

  873d421 gencores: testing crossbar fix for #1401
  7711721 saftlib: added plausibility for arguments
  7d4512a saftlib: enhanced input2event functions
  4f88aa8 build: prevent make from removing firmware .bin files
  f313244 ftm-ctl: fixed bug when setting preptime

• Congrats!

Remarks on Firmware Identification

See here for directions on how-to inspect Firmware of Soft-CPUs.

Remarks on VME or other TR Form Factors

The example above is for the SCU, where the new image is written to the flash chip via the PCIe host bus bridge. On other form factors like VME, consider using USB instead.

Connect the TR to your host system (PC, notebook...) via a USB cable. Use dev/ttyUSB0 instead of dev/wbm0.

Troubles

Erasing Flash Chip Takes Really Long with PEXARIA

Probabaly, the non-volatile config in the SPI-Flash got lost. Try

eb-config-nv < device > 10 4

This needs a power cycle to become effective.

Force Flashing in Case of Trouble

ArriaII

(VETAR2A/VETAR2A-EE-BUTIS/SCU2/SCU3)

eb-flash -s 0x40000 -w 3 dev/wbm{ID} {DEV_NAME}.rpd

ArriaV (PEXARIA...)

(PEXP/PEXARRIA5/PMC/MICROTCA/EXPLODER5)

eb-flash -s 0x10000 -w 3 dev/wbm{ID} {DEV_NAME}.rpd

Known "bad" device <> host combinations

• PMC with MENA20X (flashing/eb-flash)

-- DietrichBeck - 3 Nov 2021