AMC Testing and Commissioning Guide

Required components for each uTCA device

• SFP (green/purple)
• LC cable and a white rabbit switch (Recommended: RUNNING AT LEAST VERSION 4.1, if you can afford this)
• Quartus (Version 16.0.0 Build 211 was used for this guide)
• USB cable (micro 2.0)
• Power supply (if you don't use a crate)
• Xilinx programmer (Platform Cable USB II, DLC10)
• Altera USD blaster + PROMO11 adapter
• 5x LEMO cable(s)
• N.A.T. NAMC EXT card (exposed backplane IOs)
• A second Timing Receiver with 5 LVTTL IOs (AMC, Exploder5, ...)
• MicroTCA crate ( ToDo: Add desired crate here )
• Libera crate ( ToDo: Add desired crate here )
• $dev is a placeholder for dev/ttyUSB[X] AND dev/wbm[X] - you have to test both interfaces!
• $saftlib-dev is a placeholder for tr[X], baseboard[X], ...

Additional Resources

• BEL projects building HOW-TO

Preparation

1. Check out bel_projects
2. $ git checkout https://github.com/GSI-CS-CO/bel_projects.git
3. $ cd bel_projects
4. $ git checkout tag doomsday-4.0.4
5. $ ./fix-git.sh
6. $ ./install-hdlmake.sh
7. $ make
8. (additional) $ make driver-install
9. (additional) $ make etherbone-install
10. (additional) $ make saftlib-install
11. (additional) Use insmod to load the drivers oder restart

Important Steps

Programming the CPLD

1. Turn on power
2. Use Cosylab's boxed header to micro USB adapter and connect via JTAG (xilinx programmer)
3. Run ISE
4. Load project bel_projects/syn/gsi_microtca/cpld/microtca_prog.xise
5. Process Menu => Implement Top Module
6. Tools Menu => Impact
7. Double-click Boundary Scan
8. Control-I => microtca_prog.jed
9. Operations Menu => Program
10. Turn power off

Programming the FPGA

1. Turn power on
2. Connect JTAG via PROMO11 adapter
3. Open Quartus and program the FPGA (or use the command line: quartus_pgm -c 1 -m jtag -o 'p;microtca.sof')
4. (optional) Write this bit-stream into the SPI flash: eb-flash $dev microtca.rpd (in case USB and flash chip are already programmed)

Programming the USB Chip

1. Run 'make' in bel_projects/ip_cores/etherbone-core/hdl/eb_usb_core
2. Make sure, that no other timing receiver is attached by USB
3. Erase the USB controller (as root): ./flash-fx2lp.sh -E
4. Program the USB controller (as root): ./flash-fx2lp.sh
5. Turn power off and on

Configuring (and programming) the SPI Flash Chip

1. Program the FPGA again: quartus_pgm -c 1 -m jtag -o 'p;microtca.sof'
2. Configure the SPI flash chip: eb-config-nv $dev 10 4
3. [at room temperature] Write the bit-stream into the SPI flash: eb-flash $dev microtca.rpd (in case FPGA is not persistently programmed
4. Turn power off and on
5. [at high temperature > crate without cooling] Write the bit-stream into the SPI flash again: eb-flash $dev microtca.rpd
6. Repeat 5. three times

Formatting the WR EEPROM (MAC address is stored here)

1. Go to bel_projects/ip_cores/wrpc-sw/tools
2. Run "make"
3. ./eb-w1-write $dev 0 320 < sdb-wrpc.bin

Check White Rabbit

1. eb-console $dev
2. Type in "gui", white rabbit status should be: locked and calibrated
3. Remove and apply the fiber cable five times and make sure WR locks again
4. Press ESC to quit
5. Power cycle the device and repeat step #3 -> Watch the WR LEDs (the receiver should synchronize again)

1. Remove fiber cable
2. Type in "mode master", node should be able to lock the PLL and become a master
   Quit console
3. (optional) Test status via Saftlib $ saft-ctl $saftlib-dev -s

Check External Reference Clock

• Provide a 10MHz clock (i.e. from a White Rabbit switch)
• Drive IO_CLKIN_EN to High $ saft-io-ctl $saftlib-dev -n IO_CLKIN_EN -d 1 (Note: In the foreseeable future it will work like this: $ saft-io-ctl $saftlib-dev -n IO5 -q 1)
• Open the White Rabbit console again and type in: mode master
• Check if WR is really locked (LED, eb-console -> GUI, generate clocks, ...)
• Additionally check if the system works with a 20MHz input clock too
• (optional) Turn off master mode (mode slave)
• (optional) Generate a clock on this and on another receiver and compare them

Check EEPROM and set MAC

1. Run eb-console $dev
2. (optional) Set MAC address for the device #xy: <<mac setp aa:bb:cc:dd:ee:ff, Control-C
3. Check given MAC address
4. Turn power off and on
5. Run eb-console $dev
6. Type in "mac", you should see the previously entered MAC address
7. If you don't have a DHCP server, you can set an ip address by "ip set 192.168.100.xyz"

Check IOs

Front Panel IOs

1. Get a second uTCA timing receiver (or a receiver with at least 5 LVTTL IOs)
2. Connect IO1 (device #1) to IO1 (device #2) and so on... [IO1,IO2,IO3,IO4,IO5]
3. Device #1: Turn output enable for the IOs on $ saft-io-ctl $saftlib-dev -n IO{1,2,3,4,5} -o 1
4. Device #1: Start a clock on each IO $ saft-clk-gen $saftlib-dev -n IO{1,2,3,4,5} -f 10000000 0 (will generate a 10 MHz clock)
5. Device #2: Snoop inputs $ saft-io-ctl $saftlib-dev -s
6. Measure jitter and signal quality with a scope (FTRN specification should be matched)

Slew Rate

1. Measure the slew rate with a scope (for EVERY IO), should be equal to Exploder5

IO Load Test

1. Drive every IO with termination
2. $ saft-io-ctl $saftlib-dev -n IO{1,2,3,4,5,...} -o 1 -t 0/1 -d 1
3. Measure levels/voltage with a scope

Termination Test

1. Drive every IO with and without termination
2. Make sure that the switchable termination works

IO Bandwidth

1. Generate a 200MHz clock (saft-clk-gen) and measure the clock with a scope
2. Generate a 125MHz clock (with an additional receiver or reference device) and measure it with the ECA (saft-io-ctl snoop mode)

Output Enable Test

1. Drive every IO with and without output enable
2. Make sure you don't see a level change when output enable is turned off
3. Check the LEDs, they should indicate every activity and the output enable status

uTCA 4 Clocks A B C D

• Use the modified debug card (N.A.T. GmbH NAMC-EXT V1.5), which connects/loops the lines
• Turn on a clock and check it (siehe "Front Panel IOs")
• Measure jitter and signal quality with a scope (FTRN specification should be matched)

uTCA 4 PORT 17-20 Triggers

• Use the modified debug card (N.A.T. GmbH NAMC-EXT V1.5), which connects/loops the lines
• Turn on a clock and check it (siehe "Front Panel IOs")
• Measure jitter and signal quality with a scope (FTRN specification should be matched)

Libera Triggers

• Use the modified debug card (N.A.T. GmbH NAMC-EXT V1.5), which connects/loops the lines
• Turn on a clock and check it (siehe "Front Panel IOs")
• Measure jitter and quality with a scope

Check PCIe

1. Try the following tools:
   
   • eb-console $dev
   • eb-info $dev
   • eb-ls $dev

2. Write to internal shared ram and read it back:

   • Get the LM32 shared ram address by eb-ls $dev
   • Example output: 3.2 0000000000000651:81111444 84000 LM32-RAM-Shared
   • Create a dummy file (which will be written into the ram): dd if=/dev/urandom of=foo bs=4k count=1;
   • Write dummy file to the lm32 shared ram: eb-put $dev 0x84000 foo
   • Get the data from the shared ram: eb-get $dev 0x84000/4096 bar
   • Compare both files: cmp foo bar
   • Both files should contain the same data
   • Repeat this test in a loop... (recommended: 10 minutes)

Check PCIe interrupts

1. Create your own schedule (or use the attached one -> schedule.txt)
2. Start snooping for events: $ saft-ctl $saftlib-dev snoop 0 0 0 -x
3. Inject events: $ saft-dm $saftlib-dev -fp -n 1 schedule.txt
4. Verify that you got all events

Check LEDs

1. $ saft-pps-gen $saftlib-dev -s
2. Leave the application running and check all LEDs

Check Buttons and Hex Switches

1. $ saft-io-ctl $saftlib-dev -s
2. Start rotating switches (HSF1) => Check output of saft-io-ctl
3. Start pushing buttons (PBF1) => Check output of saft-io-ctl

Check Display

1. cd bel_projects/tools/display
2. make
3. ./simple-display dev/ttyUSBx -s "Hello World!" -d 2

Check debug port HPLA

1. To be defined

Check OneWire Devices

1. Go to: tools/commissioning/onewire-scanner
2. make
3. run application: ./onewire-scanner dev/ttyUSB0

Check OneWire Devices (eb-mon)

1. Check device with eb-mon

AMC/uTCA Stuff

Program MMC flash

1. Clone the firmware repository: $ git clone https://github.com/GSI-CS-CO/mtca_mmc_firmware.git
2. Read anf follow README.md file
3. Optional: LPC2136_FreeRTOS_CoreIPM.bin

Check MMC USB

1. minicom -b 115200 -D /dev/ttyUSB0
2. Press "i"

Check MCH and FTRN FRU info and sensors

1. Connect to the MCH (USB: minicom -b 19200 -D /dev/ttyACM0)

Check PCIe Devices

1. Run $ show_fru

Check FRU Information

1. Check FRU information: $ show_fruinfo 8

Check Sensors

1. Check sensors: $ show_sensorinfo 8

Check PCIe Speed

1. Run: $ show_link_state
2. Look for this Output: AMC 4 Port 4 is PCIe - x1 - 2,5 GT/s
3. Verify it comes from ID 8: $ show_link_state 8 4 => AMC 4 Port 4 is PCIe - x1 - 2,5 GT/s

Stop and Start Unit

1. $ shutdown 8
2. $ fru_start 8

IPMI FRU Test

1. Check FRU and sensors via IPMI tools

Ideas for FAT

General Ideas

• Generate a different frequency (saft-clk-gen) on every IO and make sure WR still works
• Check for crosstalk, mix inputs and outputs
• PCB delay measurement (compare to exploder5)
• Check 5V input on front panel IOs

Libera

• Check iTech "Low-Jitter-Clock", 1ps jitter should be achieved
• 24h PCIe write and read test
• Multiple modules (TR) at the same time in a crate

How-Swap (Propoal DB, to be reviewed by AH and DS)

• uTCA
  1. operate AMC FTRN in a uTCA crate; use power supply from 'White List'
  2. pull handle at FTRN
  3. check FTRN can be removed (FPGA no longer powered)
  4. remove FTRN
  5. insert another FTRN
  6. check FTRN is again fully powered
  7. (restart PCIe driver)
  8. check FTRN is reachable via 'eb-ls'
• Libera
  1. operate AMC FTRN in a Libera crate
  2. pull handle at FTRN and make sure nothing happens:
  3. check FTRN remains powered
  4. check FTRN remains reachable via 'eb-ls'
  5. release or push-back handel at FTRN
  6. check FTRN remains powered
  7. check FTRN remains reachable via 'eb-ls'