PEXP Testing and Commissioning Guide

Required components for each PEXP device

• SFP (green/purple)
• LC cable and a white rabbit switch (Recommended: RUNNING AT LEAST VERSION 4.1, if you can afford this)
• Quartus (Version 18.0.0+)
• USB cable (micro 2.0)
• Power supply (if you don't use a crate)
• Xilinx programmer (Platform Cable USB II, DLC10)
• Altera USB blaster + PROMO11 adapter with micro USB cable
• 5x LEMO cable(s)
• A second Timing Receiver with 5 LVTTL IOs (AMC, PEXP, Exploder5, ...)
• $dev is a placeholder for dev/ttyUSB[X] AND dev/wbm[X] - you have to test both interfaces!
• $saftlib-dev is a placeholder for tr[X], baseboard[X], ...

Additional Resources

• BEL projects building HOW-TO

Preparation

1. Check out bel_projects
2. $ git clone https://github.com/GSI-CS-CO/bel_projects.git
3. $ cd bel_projects
4. $ git checkout master
5. $ make
6. (optional): make pexp
7. (optional) $ make driver-install
8. (optional) $ make etherbone-install
9. (optional) $ make saftlib-install
10. (optional) Use insmod to load the drivers oder restart

Alternate

There is a Jenkins job used to build gateware image for chosen branches of the bel_projects and SAFTlib repos. Please look at Jenkins->timing->tests->test_build_timing_devices at https://builder.acc.gsi.de/jenkins/

Important Steps

Programming the CPLD

1. Turn on power
2. Use Cosylab's "boxed header plug to micro USB" adapter and connect via JTAG (xilinx programmer)
3. Run ISE
4. Load project bel_projects/syn/gsi_PEXP/cpld/PEXP_prog.xise
5. Process Menu => Implement Top Module
6. Tools Menu => Impact
7. Double-click Boundary Scan
8. Control-I => PEXP_prog.jed
9. Operations Menu => Program
10. Turn power off

Programming the FPGA via JTAG

1. Turn power on
2. Connect USB blaster to the JTAGCON1 port (use PROMO11 adapter with micro USB cable)
3. Open Quartus and program the FPGA (or use the command line: quartus_pgm -c 1 -m jtag -o 'p;PEXP.sof')
4. (optional) Write this bit-stream into the SPI flash: eb-flash $dev PEXP.rpd (in case USB and flash chip are already programmed)

Programming the USB Chip

1. Run 'make' in bel_projects/ip_cores/etherbone-core/hdl/eb_usb_core
2. Connect the USB micro cable to the USBCON1 port. Make sure, that no other timing receiver is attached by USB
3. Erase the USB controller (as root): ./flash-fx2lp.sh -E
4. Program the USB controller (as root): ./flash-fx2lp.sh
5. Turn power off and on

Configuring (and programming) the SPI Flash Chip

1. Program the FPGA again: quartus_pgm -c 1 -m jtag -o 'p;PEXP.sof'
2. Configure the SPI flash chip: eb-config-nv $dev 10 4 (ie., eb-config-nv dev/ttyUSB0 10 4)
3. [at room temperature] Write the bit-stream into the SPI flash: eb-flash $dev PEXP.rpd (in case FPGA is not persistently programmed
4. Turn power off and on
5. [at high temperature > crate without cooling] Write the bit-stream into the SPI flash again: eb-flash $dev PEXP.rpd
6. Repeat 5. three times

Formatting the WR EEPROM (MAC address is stored here)

1. Go to bel_projects/ip_cores/wrpc-sw/tools
2. Run "make"
3. ./eb-w1-write $dev 0 320 < sdb-wrpc.bin

Check EEPROM and set MAC

1. Run eb-console $dev
2. (optional) Set MAC address: mac setp aa:bb:cc:dd:ee:ff
3. Check given MAC address (if address is just set, then it can be checked only after power cycle!)
4. Turn power off and on
5. Run eb-console $dev
6. Type in "mac", you should see the previously entered MAC address
7. If you don't have a DHCP server, you can set an ip address by "ip set 192.168.100.xyz"

Check OneWire Devices

1. Go to bel_projects/tools/commissioning/onewire-scanner
2. make
3. run application: ./onewire-scanner $dev (ie., ./onewire-scanner dev/ttyUSB0)

Check OneWire Devices (eb-mon)

1. Check device with eb-mon
2. Get the serial code of temperature sensor: eb-mon $dev -w0 -b0 -f0x28
   0xbd0000091958c628
3. Get the actual temperature: eb-mon $dev -w0 -t0 -f0x28
   45.1875
4. Get the serial code of EEPROM: eb-mon $dev -w0 -b0 -f0x43
   0x89000000ba822c43

Check White Rabbit

1. eb-console $dev
2. Type in "gui", white rabbit status should be: locked and calibrated
3. Remove and apply the fiber cable five times and make sure WR locks again
4. Press ESC to quit
5. Power cycle the device and repeat step #3 -> Watch the WR LEDs (the receiver should synchronize again)

1. Remove fiber cable (TO BE CHECKED!)
2. Type in "mode master", node should be able to lock the PLL and become a master
   Quit console (CTRL+C)
3. (optional, via PCIe) Test status via Saftlib $ saft-ctl $saftlib-dev -s

Check IOs

Front Panel IOs

1. Get a second Pexp timing receiver (or a receiver with at least 5 LVTTL IOs)
2. Connect IO1 (device #1) to IO1 (device #2) and so on... [IO1,IO2,IO3,IO4,IO5]
3. Device #1: Turn output enable for the IOs on $ saft-io-ctl $saftlib-dev -n IO{1,2,3,4,5} -o 1
4. Device #1: Start a clock on each IO $ saft-clk-gen $saftlib-dev -n IO{1,2,3,4,5} -f 10000000 0 (will generate a 10 MHz clock)
5. Device #2: Snoop inputs $ saft-io-ctl $saftlib-dev -s
6. Measure jitter and signal quality with a scope (FTRN specification should be matched)

Slew Rate

1. Measure the slew rate with a scope (for EVERY IO), should be equal to Exploder5

IO Bandwidth

1. Generate a 200MHz clock (saft-clk-gen) and measure the clock with a scope
2. Generate a 125MHz clock (with an additional receiver or reference device) and measure it with the ECA (saft-io-ctl snoop mode)

IO Load Test

1. Drive every IO with termination
2. $ saft-io-ctl $saftlib-dev -n IO{1,2,3,4,5,...} -o 1 -t 0/1 -d 1
3. Measure levels/voltage with a scope
   measured 2,54V/2,82V with/without termination

Termination Test

1. Drive every IO with and without termination
2. Make sure that the switchable termination works

Output Enable Test

1. Drive every IO with and without output enable
2. Make sure you don't see a level change when output enable is turned off, output should stay at 0 voltage
3. Check the LEDs, they should indicate every activity and the output enable status

Check PCIe/PCI

1. Install the board in host or test PC ($dev should be dev/wbm[X])
2. Try the following tools:
   
   • eb-console $dev
   • eb-info $dev
   • eb-ls $dev

3. Write to internal shared ram and read it back:

   • Get the LM32 shared ram address by eb-ls $dev
   • Example output: 3.2 0000000000000651:81111444 84000 LM32-RAM-Shared
   • Create a dummy file (which will be written into the ram): dd if=/dev/urandom of=foo bs=4k count=1;
   • Write dummy file to the lm32 shared ram: eb-put $dev 0x84000 foo
   • Get the data from the shared ram: eb-get $dev 0x84000/4096 bar
   • Compare both files: cmp foo bar
   • Both files should contain the same data
   • Repeat this test in a loop... (recommended: 10 minutes)

Check PCIe/PCI interrupts

1. Create your own schedule (or use the attached one -> schedule.txt)
2. Start snooping for events: $ saft-ctl $saftlib-dev snoop 0 0 0 -x
3. Inject events: $ saft-dm $saftlib-dev -fp -n 1 schedule.txt
4. Verify that you got all event

Check LEDs

1. $ saft-pps-gen $saftlib-dev -s
2. Leave the application running and check all LEDs

Check Buttons and Hex Switches

1. $ saft-io-ctl $saftlib-dev -w # Wipe internal event table
2. $ saft-io-ctl $saftlib-dev -s
3. Start rotating switches => Check output of saft-io-ctl
4. Start pushing buttons => Check output of saft-io-ctl

Check Display

1. cd bel_projects/tools/display
2. make
3. ./simple-display dev/ttyUSBx -s "Hello World!" -d 2

Check debug port HPLA

1. The PEXP has the following test settings:
   • Hex Switch FPGA (HSF1) set to 'F' and button PBF1 not pressed: Status LEDs and user LEDs off, LVDS activity indicator BLUE LEDs off, LVDS direction indicator RED LEDs off.
   • Hex Switch FPGA (HSF1) set to 'F' and button PBF1 pressed: Status LEDs and user LEDs on, LVDS activity indicator BLUE LEDs on, LVDS direction indicator RED LEDs on.
   • Hex Switch FPGA (HSF1) set to 'E' and button PBF1 not pressed: LVDS termination pins disabled.
   • Hex Switch FPGA (HSF1) set to 'E' and button PBF1 pressed: LVDS termination pins enabled.
   • Hex Switch FPGA (HSF1) set to 'D' and button PBF1 not pressed: user LEDs display CPLD hex switch and button PBP1.
   • Hex Switch FPGA (HSF1) set to 'D' and button PBF1 pressed: user LEDs display inverted CPLD hex switch and button PBP1.
   • Hex Switch FPGA (HSF1) set to 'C' and button PBF1 not pressed: user LEDs display positions 7 downto 0 of xwb control signals for pmc master.
   • Hex Switch FPGA (HSF1) set to 'C' and button PBF1 pressed: user LEDs display positions 15 downto 8 of xwb control signals for pmc master.
   • Hex Switch FPGA (HSF1) set to 'A' and button PBF1 not pressed: Positions 15 down to 10 of shift register displayed on status LEDs, positions 9 down to 5 of shift register displayed on LVDS direction indicator LEDs (red), positions 4 down to 0 of shift register displayed on LVDS activity indicator LEDs (blue).
2. For details see bel_projects/top/gsi_pexp/control/pexp_control.vhd
3. For testing of HPLA port, modify pexp_control.vhd such that 'hpwck' and 'hpw' receive the clock signals (20MHz, 125MHz), the state of the two hex switches and the push buttons. These signals can be measured with an osziloscope at the pins 2 to 18 of the HPLA port.

    hpw(0) <= clk_125m_pllref_i;
    hpw(1) <= clk_125m_local_i;
    hpw(2) <= clk_sfp_ref_i;
    hpw(7 downto 3) <= con;
    hpw(12 downto 8) <= s_test_sel;
    hpw(13) <= con(5);
    hpw(14) <= s_test_sel(4);
    hpw(15) <= clk_20m_vcxo_i;
    hpwck  <= clk_20m_vcxo_i;