Equipment

• 3x MVLC (FRS) (labels: -0100, -0101, -0092)
  • registered on R3B VLAN
  • mvlc-0101 and mvlc-0092 fail a FW check with cmvlc
    • All (92, 100, 101) updated to firmware 0x0046.
• 3x VULOM4C (EE) (labels: -17745, -17752, -17772)
  • with CPLD patches
  • with 3b935cc3 trloii fw
• 3x VETAR2 (+spares) (labels: -vmel012t, -vmel015t, -vmel023t)
  • enigma FW
• 3x VFTX2 (<R3B) (labels: -11061, -17685, -17645)
• 3x DOFI (SFRS or EE) (labels: RA-8, RA-21, RA-22)
  • DOFI FW
• 1x DOFI (R3B) (label: RA-26)
  • borrowed Pi3b+ from EE
• 4x QSFP (EE)
• 2x MPO patch cable (same)
• 3x VME bin ( labels: R3B SOFIA2, VME006/30-32-94-00-08-69, R15C1)

Note: Number written after "-" was read from the white label attached to the module.

Documentation

Overview

• 
• All MVLCs are connected to the R3B network.
• All DOFIs are connected to the MBS network.
• All VETARs are connected to WR.
• Detailed node layouts and cabling are available in Full setup documentation r4.pdf.
• To edit documentation file or schematics, use Documentation file r4.pptx.

Node Details

Node 0 "middle rack"

• MVLC-0101 (MTL) is connected to R3B SC05 net.
• DOFI RA-21 and RA-8 were used, RA-21 has fiber connection with RA-26, RA-8 has fiber connection with RA-22.
• VETAR2A is connected WR (WRS-3/18 port 3).

Node 1 "left-side rack"

• MVLC-0100 is connected to R3B SC05 net.
• DOFI RA-22 was used, RA-22 has fiber connection with RA-8.
• VETAR2A is connected WR (WRS-3/18 port 18).

Node 2 "right-side rack"

• MVLC-0092 isconnected to R3B SC05 net.
• DOFI RA-26 was used, RA-26 has fiber connection with RA-21.
• VETAR2A is connected WR (WRS-3/18 port 4).

DOFI Configuration / DOFI Notes

Important Notes

• "Auto Apply" in the DOFI GUI must be enabled before modifying the logic matrix.
• For LEMO connections, the channel number used in the logical matrix is offset by +1 compared to the labelling on the DOFI front panel.
  • e.g: LEMO input/output 2 on the DOFI corresponds to input/output 3 in the logic matrix.
• Configuration files are attached below.

Configuration Files

The attached .dof files can be loaded either:

• directly from the DOFI GUI
• or via command-line tools.

Files:

• RA-8: ra8.dof
• RA-21: ra21.dof
• RA-22: ra22.dof
• RA-26: ra26.dof

CLI Monitoring

• A simple Python-based CLI rate monitor (dofistat.py) was implemented for DOFI monitoring, e.g. dofistat.py ra-22.
• The tool displays live scaler/rate information per second for: Fiber input/output, LEMO input/output

Location: R3B Cave C

• 3x VME bin
  • TODO: remote control
• 1x WR switch
• MBS network switch
• need R3B network switch
  • TODO
• LEMO, ECL cables
• ENV modules

Server: land@epics0.r3b.gsi.de

• cmvlc
• drasi
• trloii
• nurdlib
  • TODO: cleanup needed before MRs
• scripts and configs
• unpacker
• online analysis (pyh101/pyjsroot)

Issues (bugs)

• Nurdlib cmvlc readout on triggers 14 and 15: no timestamps in VETAR.
  • Reason: trig 14 and 15 are software-generated - no pulses on front-panels sent. (As expected.) Modules should not be read...
  • Workaround: not generating cmvlc readout for triggers 14 and 15.
  • General: need to figure out 'tags' use in nurdlib with cmvlc. (Even if, at the end, not the solution for trig 14 and 15.)

• Two entries in VETAR FIFO at high rates (3 MHz Poisson trigger)
  • Possible things to test:
    • Is it seen in all systems or just one?
    • Add a second (and third) redundant read of the FIFO count and see that they match.
    • Add a stack sleep (wait) command after e.g. the pop, such that next read if fifo count is for sure delayed. Or after all the VETAR reads. Or put the wait before the FIFO count read.
    • Do trigger requests in bursts. E.g. with a low rate, but then in rapid succession. More than two in rapid succession. See if issues appear at a certain succession rate.
    • Add a scaler to read a copy of the actual latch signal sent to the VETAR, to see if there are double-pulses.

• TRIVA works with triva_event ... --triva, but not with the drasi/example/*cmvlc*mdpp or nurdlib.
  • Not really a relevant problem for SEP.
  • Late read issue?

• SFP issue (where? MVLC Ethernet?)
  • Works with one network switch (model/brand?) but not another (model/brand?)?

• trloctrl --mux-src-scalers sometimes showing weird values and differences
  • Only VULOM in crate with mvlc-0100
    • Swapped VULOM with mvlc-0092 one, problem moved with VULOM, probably hardware related.
  • Scalers up to internal index 127 work, 128 and higher do not work
  • Suspected: FPGA (hardware) issue. (Since working/non-working aligns with the internal scaler block structure.)
  • TODO: check if MVLC firmware upgrade helped (NO)
  • VULOM serial number: 17745

Mis-features

• MVLC hostname given both on the drasi command line and nurdlib.cfg. Recipe for mistakes.
  • Possible solution: drasi gets it from nurdlib when compiled with nurdlib.
    • Drawback: hostname is in nurdlib.cfg file, which otherwise need not be updated if changing hosts.
  • Possible solution: nurdlib gets it from drasi when compiled with drasi.
    • Could be via mechanism (but fully internal, in fuser_drasi) similar to changing(overriding) parameters with nurdctrl, but happen on startup.

• drasi+cmvlc+nurdlib readout with misbehaving module needs more error handling:
  • With a VFTX2 labelled with 'readout issue', it did not work too stably (not the fault of the readout ).
  • See ~/software/mvlc/mockup_tests/mvlc-0100/fail.tcpdump
  • More error handling is needed: the node hung half-alive for hours before it decided a reset was in order, and the init write failed.

• drasi+cmvlc: Ctrl-C-ing a daq will sometimes lead to an assert_lock in cmvlc. However, it still kills the daq mode first, so it is more of a cosmetic issue.

Performing hardware cleanup (TRIVA HALT, RESET)...
Data frame with wrong type: 0xf1000030.
Supercmd reply with unexpected length: 18 != 51.
assert_lock called while we do not hold the lock. Sleep forever.

Todo

• Try bidirectional DOFI.
  • LEMO IN 1-4 on DIFO A via optical to LEMO OUT 1-4 on DOFI B and vice versa.

• mvlc-0092 to mvlc-0101 send the deadtime to MTL.

• mvlc-0092 should send trigger requests to MTL.

• VFTX2 nurdlib readout in one crate.

• More documentation (wiki?)

• Online monitoring/histogramming.

• Recable mvlc-0100 to use DOFI. (Same configuration as mvlc-0092).

• Cable one system to use diff (ECL/LVDS) signals between VULOM/TRLO II and MUPPET/DOFI.

• Open and close files (time orderer) using lwrocctrl.

• Send low-jitter signal between systems and measure using VFTX2s (sync'ed to 200 MHz clock).
  • Direct cable.
  • Using highland.

• Command-line interface for DOFI?
  • CLI rate monitor was written in Python.

Data format

Sample data from early 1970:

00000420 03e1905f 04e17491 05e1e301 06e10000 f1a00003 babababa 81c00001
00000003 babababa e3010000 905f7491

• six words of WR header
  • One word containing the White Rabbit ID (and an error marker)
  • Four words containing 16 bits of the TS each (and markers in the upper half of the word)
  • One word containing the sync value (plus a marker)

• barrier word

• a few words from the VULOM
  • tpat
  • sync
• barrier word

• VETAR
  • two words containing the timestamp (32 bits each, possibly with a weird byte order)

The FIFO count of the VETAR is asserted to be one during fetch and not written to the

Resolved

Move things down here from #OpenIssues when resolved. Perhaps delayed so as not to have to move up again if reopened...

Resolved Issues (bugs)

• VME bus error with VETAR
  • roughly one in ~5k events
  • tried NIM->TTL, no change
  • TODO: add debugging of failed stack
  • TODO: late reads?
  • The FIFO count was empty for those events.
  • FIXED: Drawing on the wisdom of the ancients (i.e. traditional R3B vulom.trlo files), I now use ACCEPT_PULSE instead of MASTER_START for the VETAR gate.
  • Actual issue: trigger was generated with 'pending' triggers, which gives no MASTER_START. Was masked by dual trigger generation, also with TPAT trigger, which generated the trigger most of the time, giving a MASTER_START.

• Drasi timesorter dropping all events.
  • BUG! Introduced Jan 27 (23b5ec92).
  • Fixed: b8461fdb.
  • Improvement: drasi CI now does some basic runsim.sh tests. (I.e., actually runs drasi a few seconds for a few different configurations for each CI pipeline.)
    • Failing example (with bug): https://git.chalmers.se/expsubphys/drasi/-/jobs/823258
    • Working example (with fix): https://git.chalmers.se/expsubphys/drasi/-/jobs/823120