Lean Implementation of new LMD file format

MbsDataFormats

Usage of buffer header
A special buffer header is used as file header. Important fields like word size, type, and endian are compatible with other buffer headers, especially MBS. It might be followed by header data.
iEndian: always 1
iType: 0x00010065 (type 101,1)
iTimeSpecSec
iTimeSpecNanoSec
iUsedWords: Number of words following buffer header.
iElements: Number of elements in file (without header, without index table)
iMaxWords: Total size of largest element
iTableOffset: uint64_t optional offset to element index table at end of file.
iOffsetSize: 4 or 8 if index table is 4 or 8 byte.

Functions

sLmdControl * fLmdAllocateControl();
Returns
Returns allocated zeroed control structure. Must be freed after Close.
Description

uint32_t fLmdPutOpen(...);
sLmdControl *Control
Control structure. Must be allocated from caller by fLmdAllocateControl(). Must be passed as first argument to all other calls. Is cleared by Open functions. Must be freed by caller.
char File
Filename
sMbsFileHeader *FileHeader
Optional file header (or LMD__STANDARD_HEADER, then allocated internally). Behind sMbsFileHeader structure arbitrary data may follow, size [words] must be in sMbsFileHeader.iUsedWords. After Open calls file header is accessible by sLmdControl.sMbsFileHeader. Internally allocated file header is freed by Close functions.
uint32_t Buffersize
Size of internal buffer to be used for write operations. If LMD__NO_BUFFER, no buffering is used and event is written directly by fLmdPutElement. When using fLmdPutBuffer function no internal buffer is needed.
uint32_t Overwrite
LMD__OVERWRITE or LMD__NO_OVERWRITE to overwrite existing file or not.
uint32_t Indextable
LMD__INDEX or LMD__NO_INDEX to create index table or not. The index table contains the offsets of all elements in the file. The table is located at file end. The offset of the table itself is in the file header. The table offsets are stored in 32 bit. If Largefile is enabled (see next) offsets are stored in 64 bit thus allowing for more than 16 GB files. Offsets values are in units of 4 bytes allowing for 16GB to be addressed with 32 bit.
uint32_t Largefile
LMD__LARGE_FILE or LMD__NO_LARGE_FILE to enable large file support for index tables. Needed only when index table is enabled and file should get larger than 16 GB.
Returns
LMD__SUCCESS
PUTLMD__FILE_EXIST
PUTLMD__OPEN_ERR
Description
Open file for write. Overwrite existing file optionally. If buffersize is greater zero, use internal buffer of this size. Otherwise write elements one per call. First write file header. If header is passed from caller, write subsequent sMbsFileHeader.iUsedWords words to file. Otherwise use internal header.

uint64_t fLmdGetBytesWritten(...);
sLmdControl *Control
Returns
Number of bytes written to file if it would be closed.

uint32_t fLmdGetSwap(...);
sLmdControl *Control
Returns
1: Data will be swapped (4 byte wise)
0: Data will not be swapped
-1: No file opened.
Can be called after fLmdGetOpen or fLmdConnectMbs.

void fLmdSetWrittenEndian(...);
sLmdControl *Control
uint32_t endian
LMD__ENDIAN_UNKNOWN, LMD__ENDIAN_LITTLE, LMD__ENDIAN_BIG
This is written in the file header and can be retrieved. Can be called after fLmdPutOpen before fLmdPutClose.

uint32_t fLmdGetWrittenEndian(...);
Endian information is written in the file header and can be retrieved. Can be called after fLmdGetOpen or fLmdGetMbsEvent.
sLmdControl *Control
Returns
LMD__ENDIAN_UNKNOWN: no info about endian
LMD__ENDIAN_LITTLE: file was written by little endian machine (Intel)
LMD__ENDIAN_BIG: file was written by big endian machine (IBM)

uint32_t fLmdPutElement(...);
sLmdControl *Control
sMbsHeader *ElementBuffer
sMbsHeader.iWords must contain the total number of words - 4.
Returns
LMD__SUCCESS
PUTLMD__TOOBIG: Element too big for buffer.
PUTLMD__EXCEED: File size exceed. Files opened with LMD__NO_LARGE_FILE and LMD__INDEX must not exceed 16GB. Current element is not written.
LMD__FAILURE
Description
Write one element to file (direct or via internal buffer).

uint32_t fLmdPutBuffer(...);
sLmdControl *Control
sMbsHeader *ElementBuffer
uint32_t NumberOfElements
Number of elements in buffer.
Returns
LMD__SUCCESS
PUTLMD__EXCEED: File size exceed. Files opened with LMD__NO_LARGE_FILE and LMD__INDEX must not exceed 16GB. Current buffer is not written.
LMD__FAILURE
Description
Write complete buffer to file. Buffer must contain NumberOfElements of elements headed by a sMbsHeader structure with correct word size. No internal buffer used.

uint32_t fLmdPutClose(...);
sLmdControl *Control
Returns
LMD__SUCCESS
LMD__CLOSE_ERR
If optional index table could not be written, file header contains no table reference.
Description
Write rest of internal buffer, if necessary, free all internal memory, write optionally index table and close file.

uint32_t fLmdGetOpen(...);
sLmdControl *Control
char Filename
sMbsFileHeader *FileHeader
Optional file header (or LMD__INTERNAL_HEADER, then allocated internally). After Open calls file header is accessible by sLmdControl.sMbsFileHeader. Data of file header, if any, is accessiblae by sLmdControl.cHeader. sLmdControl.sMbsFileHeader.iUsedWords is the size of this data which is never swapped. Internally allocated file header is freed by Close functions.
uint32_t Bytes
Internal buffer size. LMD__NO_BUFFER means not to use internal buffer. This must be specified when fLmdGetBuffer calls are made afterwards.
uint32_t Indextable
LMD__INDEX or LMD__NO_INDEX to use index table or not. Needed for random element read. Option for buffer read.
Returns
LMD__SUCCESS
GETLMD__NOFILE
File not found.
GETLMD__NOLMDFILE
File is shorter than file header or file header type is wrong.
Description
Open file for reading. If header memory is passed from caller, use it to read header from file, otherwise use internal memory. Allocate internal memory to read optionally iUsedWords from file. This memory can be accessed by pointer sLmdControl.cHeader. Note that internal memory is freed by Close functions.

uint32_t fLmdGetElement(...);
sLmdControl *Control
uint32_t Index
LMD__NO_INDEX to read events sequentially without using index table. If starting from 1, largest sLmdControl.iOffsetEntries , read random event. File must have index table and must be opened with LMD__INDEX flag.
sMbsHeader **event
Returns pointer to event, null if no more or failure.
Returns
LMD__SUCCESS
GETLMD__NOBUFFER: no internal buffer
GETLMD__NOMORE: no more elements
GETLMD__EOFILE: end of file
GETLMD__TOOBIG: event does not fit into buffer
GETLMD__OUTOF_RANGE: event index out of range
GETLMD__SIZE_ERROR: mismatch between size in file table and element header

Description

uint32_t fLmdGetBuffer(...);
sLmdControl *Control
sMbsHeader *ElementBuffer
Buffer of size Bytes to read elements from file. Buffer size must be at least sLmdControl.sMbsFileHeader.iMaxWords large to store largest element in file.
uint32_t Bytes
Size of buffer
uint32_t *Elements
Returns number of elements in buffer.
uint32_t *Bytes
Returns number of bytes used in buffer.
Returns
LMD__SUCCESS
GETLMD__EOFILE
GETLMD__NOLMDFILE: Size derived from index table was wrong.
LMD__FAILURE: Internal buffer to small to store fragment.
Description
User buffer is filled with complete elements. Fragment at end is stored in internal buffer and copied on top of user buffer with next call.

int32_t fLmdReadBuffer(...);
sLmdControl *Control
char *Buffer
Buffer to be filled by Bytes bytes.
uint32_t Bytes
Size of buffer
Returns
Number of bytes read.
Description
Raw buffer read. Does not scan buffer for elements. Elements may be fragments.

uint32_t fLmdGetClose(...);
sLmdControl *Control
Returns
LMD__SUCCESS
LMD__CLOSE_ERR
Description
Close file, free internal memory.

uint32_t fLmdConnectMbs(...);
sLmdControl *Control
char *Nodename
int Port
PORT__TRANS: Port number of MBS transport (6000).
PORT__STREAM: Port number of MBS stream server (6002).
uint *Buffersize
Address to receive number of bytes caller has to allocate for buffer passed to fLmdGetMbsBuffer. If LMD__GET_EVENTS, internal buffer is used for fLmdGetMbsEvent calls.
Returns
LMD__SUCCESS
LMD__FAILURE
Could not connect to MBS transport.
Description
Connects to MBS transport.

uint32_t fLmdInitMbs(...);
sLmdControl *Control
char *Nodename
int Buffersize
Buffer size in bytes needed to get buffers from transport..
int StreamBuffers
Must be 1.
int Streams
Must be 0.
int Port
Port number: MBS transport is PORT__TRANS (6000), stream server is PORT__STREAM (6002).
int timeout
Timeout in seconds for TCP reads.
Returns
LMD__SUCCESS
LMD__FAILURE
StreamBuffers greater one (would mean event spanning) or Streams greater zero (would mean MBS is not in DABC mode).
Description
This call is for f_evt_get_open when it has already connected to MBS transport. When it finds that number of streams is zero, it must call fLmdInitMbs. In sLmdControl the address of the s_tcpcomm must be set. Internal buffer is used for fLmdGetMbsEvent calls.

uint32_t fLmdGetMbsEvent(...);
sLmdControl *Control
sMbsHeader **event
Pointer to next element (NULL if no more).
Returns
LMD__SUCCESS
LMD__TIMEOUT
LMD__FAILURE
Description
Reads next buffer from MBS transport if necessary. Returns pointer to next event.

uint32_t fLmdGetMbsBuffer(...);
sLmdControl *Control
sMbsBufferHeader *EventBuffer
Buffer of size Bytes to read buffers from MBS. Buffer size must be as returned by fLmdConnectMbs.
uint32_t Bytes
Size of buffer. Buffer size must be as returned by fLmdConnectMbs.
uint32_t *Elements
Returns number of elements in buffer.
uint32_t *Bytes
Returns number of bytes used in buffer.
Returns
LMD__SUCCESS
LMD__TIMEOUT
LMD__FAILURE
Mostly if MBS has shut down.
Description
User buffer is filled with data from MBS transport.

uint32_t fLmdCloseMbs(...);
sLmdControl *Control
Returns
LMD__SUCCESS
Description
Close connection, free internal memory.

void fLmdPrintBufferHeader(...);
uint32_t Verbose
1: print, 0: silent
sMbsBufferHeader *BufferHeader
Description
Print buffer header when Verbose is set.

void fLmdPrintFileHeader(...);
uint32_t Verbose
1: print, 0: silent
sMbsFileHeader *FileHeader
Description
Print file header when Verbose is set.

void fLmdPrintHeader(...);
uint32_t Verbose
1: print, 0: silent
sMbsHeader *Header
Description
Print header when Verbose is set.

void fLmdPrintEvent(...);
uint32_t Verbose
1: print, 0: silent
sMbsEventHeader *EventHeader
Description
Print event header when Verbose is set.

void fLmdPrintControl(...);
uint32_t Verbose
1: print, 0: silent
sLmdControl *Control
Description
Print control structure, file header, and current element header when Verbose is set.

Support in f_evt

The standard API f_evt is able to connect to MBS transport in DABC mode or open a file with new format for read. The new format can also be written, by specifying number of streams to 0 in the open call.

-- HansEssel - 30 Oct 2007
Topic revision: r13 - 2008-10-15, HansEssel
 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding GSI Wiki? Send feedback
Imprint (in German)
Privacy Policy (in German)