Timing Messages

Format of a Timing Message

Timing messages is a term describing the input to a Timing Receiver (TR) that possibly leads to generation of a so-called timing event depending on the local configuration of the TR.

Figure: Message Format. Digits denote the number of bits.

The figure above depicts the format of a message. All data are big endian.

1. WB Address: Wishbone address to which the data shall be written, typically the address of the ECA unit (see Documents --> ECA).
2. Payload (256 bit)

The payload contains

1. EventID (64 bit) - used for indexing of control values
2. Param (64 bit) - an additional parameter with event specific meaning. For all standard Event Numbers (EVTNO, see below) this field carries the following information
   1. 22 bit BPCID (Beam Production Chain ID)
   2. 42 bit BPCTS (Beam Production Chain Time Stamp)
3. Reserved (32/0 bit)
4. TEF (32 bit) - Timing Extension Field containing data relevant for internal processing
5. Timestamp (64 bit)
   • in units of 1ns since 1 January 1970 (release "Balloon" and later)
   • in units of 8ns clock cycles since 1 January 1970 (deprecated: prior to release "Balloon")

The EventID contains

1. 4bit FID (Format ID) - In case the format of the EventID changes in the future, this field serves to distinguish between different formats.
   • value "0": format used up to September 2017, see image
   • value "1": format used from October 2017, see image, CCT protocol
2. 12bit GID (Group ID) - Identifies a group of equipment, such as a synchrotron or a transfer line. Replaces the previous "machine ID". Grouping of equipment should follow the principle "what happens together, belongs together". A specific equipment can be member of more than one group, see here.
3. 12bit EVTNO (Event Number) - Specifies a command to be executed such as "START_CYCLE" or "START_RAMP".
4. 4bit Flags - see below for meaning of flag bits
5. 12bit SID (Sequence ID) - A sequence is analogous to the concept of a "virtual accelerator". Remark: We need > 2000 settings for therapy-like operation.
6. 14bit BPID (Beam Process ID) - A beam process defines a process like a ramp that must not be interrupted. We need > 5000 settings for therapy-like operation.
7. 6bit (eventIdAttributes) - see below for meaning

Flags

1. 1 bit Beam-in - marks type of beam process this message belongs to. TRUE: "Beam-in", FALSE: "Beam-out"
2. 1 bit BPC-Start - marks the start of the execution of a Beam Production Chain; only used for CMD_SEQ_START. TRUE: "A new BPC starts", FALSE: "no start of a new BPC". Remark: The meaning of this bit was changed from 'Reserved' to 'BPC-Start' as a consequence of a meeting on 28 May 2020. This was considered a minor change and it was decided not to increment the FID.
3. 2 bit (Reserved)

eventIdAttributes - the meaning of this field depends on the event number

• EvtNo 0d350..0d354, 0x15e..0x162 (*)
  • synchronization SIS18 <-> UNILAC
  • see here
  1. 1 bit - request virtual accelerator but without beam at UNILAC (intermediate solution for Retrofitting in 2018 ...)
  2. 4 bit - number of virtual accelerator requested at UNILAC (intermediate solution for Retrofitting in 2018 ...)
• EvtNo 0d000..0d255, 0x00..0xff (**)
  • four bits are mapped to bits 12..15 of MIL Timing telegrams, except for EvtNo 0d200..208 (0xc8..d0) 0d224..228 (0xe0..e4)and 0d255(0xff), where bits 12..15 are treated differently
  • see here
  • the four bits 0..3 of the EvtId are directly mapped to bits 12..15 of the MIL telegram (for the relevant EvtNo, see above)
  • informative: at some time in the past, the meaning of these four bits has been defined as follows
  1. 1 bit - 'high current flag' (DB 11/2024: I am not sure if this bit is actually used)
  2. 1 bit - 'no beam flag'
  3. 1 bit - 'high-brho flag'
  4. 1 bit - 'reserved'

White Rabbit Pulszentrale @ UNILAC

In Summer 2020, a first solution for White Rabbit @ UNILAC has been implemented, see here. In principle, it uses the same message format as the FAIR Data Master, although the semantics is a bit different.

The payload contains

1. EventID (64 bit) - used for indexing of control values
2. Param (64 bit) - now set to 0x0 (a mapping used previously, <= 2024, is now deprecated
3. Reserved (32/0 bit) - see above
4. TEF (32 bit), see above
5. Timestamp (64 bit) - see above

The EventID contains

1. 4bit FID (Format ID), see above
2. 12bit GID (Group ID), UNILAC timing sections 1..7 are coded into specific group numbers, see here
3. 12bit EVTNO (Event Number) - only 'old' event numbers 0..255 are used, see here
4. 4bit Flags - unused
5. 12bit SID (Sequence ID) - virtual accelerator 0..15
6. 14bit BPID (Beam Process ID) - unused
7. 6bit eventIdAttributes (see above)

See here.

Deprecated Message Formats

Format 0x0

Ethernet Frame Containing a Timing Message

A Timing Message from the Data Master to the TR, the message has to be sent across the timing network. The following has to be considered for calculating the size of a Ethernet frame including one or more Timing Messages.

1. 4 byte Wishbone address + 32 byte Timing Message (see above).
2. 8 byte Etherbone header
3. 58 byte network header (30 Ethernet + 20 IP + 8 UDP)
4. 8 byte interframe gap

An Ethernet frame including one Timing Message thus has a length of 4+32+8+58+8 = 110 byte. For each additional Timing Message, the size of the frame increases by 8+4+32 = 44 byte.

The actual amount to be transmitted through the network will be increased by a factor of 3-4 due to Forward Error Correction (FEC).

Number of Messages per Time Interval

(This disucssion might move to a different Wiki topic.)

Assumptions The following is a worst case estimate based on the following assumptions.

• The available average bandwidth for real-time networking corresponds to a rate of 100 MBit/s, see here.
• One Ethernet frame (without FEC) only contains one message.
• The interval used by the data master to plan ahead of the actual time is 500us - "ahead interval".
• FEC increases the amount of data to be sent by a factor of 3.5.

Limits The following must be considered.

• A message scheduled can not be revoked.
• The latency by which the data master can react to external information equals the "ahead interval".
• The "ahead interval" includes limits imposed by bandwidth, latency of the network, processing of FEC (at the data master) and deFEC (at the nodes) and the processing time at the nodes required for event generation on-time.
• The "ahead interval" only includes top down traffic. It does not include the time it would take to send some information via the "back channel" from the node to the data master.
• The "ahead interval" has a fixed value. Changing that value changes the latency time in which the timing system can react on external events. Features such as the machine protection system or the interlock system might rely on a known latency of the timing system.
• The data master master must respect the maximum bandwidth rate within the "ahead interval". If the bandwidth rate is 100 MBit/s and the ahead interval equals 500us, then the data master must not send more than 500 * 100 "bit" to the network within the ahead interval. The number of bits equals the gross number of bits including all headers, preamble, interframe gap,... .

Examples

• Messages per "ahead interval"
  • max rate 100 MBit/s
  • ahead interval 500us ==> 50,000 bits
  • number of bits to be sent is 110 byte * 8 (conversion to bit) * 3.5 (factor required by FEC) = 3080 bits.
  • number of message per "ahead interval" 50000 bit / 3080 bit = 16.
• "Ahead interval" as a function of message number
  • assume 60 message must be sent "at once" (meaning: within the "ahead interval")
  • number of bits to be sent 60 * 3080 bit = 184800 bit.
  • "Ahead interval" = 184000 bit / (100 Bit/us) = 1848 us = 1.8ms.

Remarks

• As shown above, the calculation above can be used in both directions. Either the timing system can be accommodate short latency time XOR it can accommodate a high number of "simultaneous" messages by increasing the value of the "ahead interval".
• The idea is to send more than one message in one Ethernet frame. An additional message adds only 44 byte. However, it is not at all clear what the maximum Ethernet frame size will be when FEC is implemented. To be on the safe side: An increase of the effective "bandwidth" by a factor of 2 is too optimistic. Maybe an increase by 50% can be achieved. But: This is all not implemented, not existing and requires the fulfillment of certain conditions outside the control of CSCO. To be on the safe side, don't count on this!
• Additional messages: If the full bandwidth would be used, it would be impossible to guarantee real-time delivery of additional exceptional messages like "post-mortem message" into the data stream (Either they can't be inserted into the data stream and would be pending until the amount of traffic for timing messages decreases, or timing messages need to be delayed and can not longer be delivered to the front-ends in real-time. In other words: If the control system foresees sending messages such as "post mortem" they must either be included in the schedule as "place holder" already at the time of the schedule planning or a certain amount of the total bandwidth must be reserved.)

-- DietrichBeck - 05 Nov 2024