Lean Implementation of new LMD file format

MbsDataFormats

Usage of buffer header

A special buffer header is used as file header. Important fields like word size, type, and endian are compatible with other buffer headers, especially MBS. It might be followed by header data.

iEndian: always 1
iType: 0x00010065 (type 101,1)
iTimeSpecSec
iTimeSpecNanoSec
iUsedWords: Number of words following buffer header.
iElements: Number of elements in file (without header, without index table)
iMaxWords: Total size of largest element
iTableOffset: uint64_t optional offset to element index table at end of file.
iOffsetSize: 4 or 8 if index table is 4 or 8 byte.

Functions

sLmdControl * fLmdAllocateControl();

Returns

Returns allocated zeroed control structure. Must be freed after Close.

Description

uint32_t fLmdPutOpen(...);

sLmdControl *Control

Control structure. Must be allocated from caller by fLmdAllocateControl(). Must be passed as first argument to all other calls. Is cleared by Open functions. Must be freed by caller.

char File

Filename

sMbsFileHeader *FileHeader

Optional file header (or LMD__STANDARD_HEADER, then allocated internally). Behind sMbsFileHeader structure arbitrary data may follow, size [words] must be in sMbsFileHeader.iUsedWords. After Open calls file header is accessible by sLmdControl.sMbsFileHeader. Internally allocated file header is freed by Close functions.

uint32_t Buffersize

Size of internal buffer to be used for write operations. If LMD__NO_BUFFER, no buffering is used and event is written directly by fLmdPutElement. When using fLmdPutBuffer function no internal buffer is needed.

uint32_t Overwrite

LMD__OVERWRITE or LMD__NO_OVERWRITE to overwrite existing file or not.

uint32_t Indextable

LMD__INDEX or LMD__NO_INDEX to create index table or not. The index table contains the offsets of all elements in the file. The table is located at file end. The offset of the table itself is in the file header. The table offsets are stored in 32 bit. If Largefile is enabled (see next) offsets are stored in 64 bit thus allowing for more than 16 GB files. Offsets values are in units of 4 bytes allowing for 16GB to be addressed with 32 bit.

uint32_t Largefile

LMD__LARGE_FILE or LMD__NO_LARGE_FILE to enable large file support for index tables. Needed only when index table is enabled and file should get larger than 16 GB.

Returns

LMD__SUCCESS
PUTLMD__FILE_EXIST
PUTLMD__OPEN_ERR

Description

Open file for write. Overwrite existing file optionally. If buffersize is greater zero, use internal buffer of this size. Otherwise write elements one per call. First write file header. If header is passed from caller, write subsequent sMbsFileHeader.iUsedWords words to file. Otherwise use internal header.

uint64_t fLmdGetBytesWritten(...);

sLmdControl *Control

Returns

Number of bytes written to file if it would be closed.

uint32_t fLmdGetSwap(...);

sLmdControl *Control

Returns

1: Data will be swapped (4 byte wise)
0: Data will not be swapped
-1: No file opened.
Can be called after fLmdGetOpen or fLmdConnectMbs.

void fLmdSetWrittenEndian(...);

sLmdControl *Control

uint32_t endian

LMD__ENDIAN_UNKNOWN, LMD__ENDIAN_LITTLE, LMD__ENDIAN_BIG
This is written in the file header and can be retrieved. Can be called after fLmdPutOpen before fLmdPutClose.

uint32_t fLmdGetWrittenEndian(...);

Endian information is written in the file header and can be retrieved. Can be called after fLmdGetOpen or fLmdGetMbsEvent.

sLmdControl *Control

Returns

LMD__ENDIAN_UNKNOWN: no info about endian
LMD__ENDIAN_LITTLE: file was written by little endian machine (Intel)
LMD__ENDIAN_BIG: file was written by big endian machine (IBM)

uint32_t fLmdPutElement(...);

sLmdControl *Control

sMbsHeader *ElementBuffer

sMbsHeader.iWords must contain the total number of words - 4.

Returns

LMD__SUCCESS
PUTLMD__TOOBIG: Element too big for buffer.
PUTLMD__EXCEED: File size exceed. Files opened with LMD__NO_LARGE_FILE and LMD__INDEX must not exceed 16GB. Current element is not written.
LMD__FAILURE

Description

Write one element to file (direct or via internal buffer).

uint32_t fLmdPutBuffer(...);

sLmdControl *Control

sMbsHeader *ElementBuffer

uint32_t NumberOfElements

Number of elements in buffer.

Returns

LMD__SUCCESS
PUTLMD__EXCEED: File size exceed. Files opened with LMD__NO_LARGE_FILE and LMD__INDEX must not exceed 16GB. Current buffer is not written.
LMD__FAILURE

Description

Write complete buffer to file. Buffer must contain NumberOfElements of elements headed by a sMbsHeader structure with correct word size. No internal buffer used.

uint32_t fLmdPutClose(...);

sLmdControl *Control

Returns

LMD__SUCCESS
LMD__CLOSE_ERR
If optional index table could not be written, file header contains no table reference.

Description

Write rest of internal buffer, if necessary, free all internal memory, write optionally index table and close file.

uint32_t fLmdGetOpen(...);

sLmdControl *Control

char Filename

sMbsFileHeader *FileHeader

Optional file header (or LMD__INTERNAL_HEADER, then allocated internally). After Open calls file header is accessible by sLmdControl.sMbsFileHeader. Data of file header, if any, is accessiblae by sLmdControl.cHeader. sLmdControl.sMbsFileHeader.iUsedWords is the size of this data which is never swapped. Internally allocated file header is freed by Close functions.

uint32_t Bytes

Internal buffer size. LMD__NO_BUFFER means not to use internal buffer. This must be specified when fLmdGetBuffer calls are made afterwards.

uint32_t Indextable

LMD__INDEX or LMD__NO_INDEX to use index table or not. Needed for random element read. Option for buffer read.

Returns

LMD__SUCCESS
GETLMD__NOFILE
File not found.
GETLMD__NOLMDFILE
File is shorter than file header or file header type is wrong.

Description

Open file for reading. If header memory is passed from caller, use it to read header from file, otherwise use internal memory. Allocate internal memory to read optionally iUsedWords from file. This memory can be accessed by pointer sLmdControl.cHeader. Note that internal memory is freed by Close functions.

uint32_t fLmdGetElement(...);

sLmdControl *Control

uint32_t Index

LMD__NO_INDEX to read events sequentially without using index table. If starting from 1, largest sLmdControl.iOffsetEntries , read random event. File must have index table and must be opened with LMD__INDEX flag.

sMbsHeader **event

Returns pointer to event, null if no more or failure.

Returns

LMD__SUCCESS
GETLMD__NOBUFFER: no internal buffer
GETLMD__NOMORE: no more elements
GETLMD__EOFILE: end of file
GETLMD__TOOBIG: event does not fit into buffer
GETLMD__OUTOF_RANGE: event index out of range
GETLMD__SIZE_ERROR: mismatch between size in file table and element header

Description

uint32_t fLmdGetBuffer(...);

sLmdControl *Control

sMbsHeader *ElementBuffer

Buffer of size Bytes to read elements from file. Buffer size must be at least sLmdControl.sMbsFileHeader.iMaxWords large to store largest element in file.

uint32_t Bytes

Size of buffer

uint32_t *Elements

Returns number of elements in buffer.

uint32_t *Bytes

Returns number of bytes used in buffer.

Returns

LMD__SUCCESS
GETLMD__EOFILE
GETLMD__NOLMDFILE: Size derived from index table was wrong.
LMD__FAILURE: Internal buffer to small to store fragment.

Description

User buffer is filled with complete elements. Fragment at end is stored in internal buffer and copied on top of user buffer with next call.

int32_t fLmdReadBuffer(...);

sLmdControl *Control

char *Buffer

Buffer to be filled by Bytes bytes.

uint32_t Bytes

Size of buffer

Returns

Number of bytes read.

Description

Raw buffer read. Does not scan buffer for elements. Elements may be fragments.

uint32_t fLmdGetClose(...);

sLmdControl *Control

Returns

LMD__SUCCESS
LMD__CLOSE_ERR

Description

Close file, free internal memory.

uint32_t fLmdConnectMbs(...);

sLmdControl *Control

char *Nodename

int Port

PORT__TRANS: Port number of MBS transport (6000).
PORT__STREAM: Port number of MBS stream server (6002).

uint *Buffersize

Address to receive number of bytes caller has to allocate for buffer passed to fLmdGetMbsBuffer. If LMD__GET_EVENTS, internal buffer is used for fLmdGetMbsEvent calls.

Returns

LMD__SUCCESS
LMD__FAILURE
Could not connect to MBS transport.

Description

Connects to MBS transport.

uint32_t fLmdInitMbs(...);

sLmdControl *Control

char *Nodename

int Buffersize

Buffer size in bytes needed to get buffers from transport..

int StreamBuffers

Must be 1.

int Streams

Must be 0.

int Port

Port number: MBS transport is PORT__TRANS (6000), stream server is PORT__STREAM (6002).

int timeout

Timeout in seconds for TCP reads.

Returns

LMD__SUCCESS
LMD__FAILURE
StreamBuffers greater one (would mean event spanning) or Streams greater zero (would mean MBS is not in DABC mode).

Description

This call is for f_evt_get_open when it has already connected to MBS transport. When it finds that number of streams is zero, it must call fLmdInitMbs. In sLmdControl the address of the s_tcpcomm must be set. Internal buffer is used for fLmdGetMbsEvent calls.

uint32_t fLmdGetMbsEvent(...);

sLmdControl *Control

sMbsHeader **event

Pointer to next element (NULL if no more).

Returns

LMD__SUCCESS
LMD__TIMEOUT
LMD__FAILURE

Description

Reads next buffer from MBS transport if necessary. Returns pointer to next event.

uint32_t fLmdGetMbsBuffer(...);

sLmdControl *Control

sMbsBufferHeader *EventBuffer

Buffer of size Bytes to read buffers from MBS. Buffer size must be as returned by fLmdConnectMbs.

uint32_t Bytes

Size of buffer. Buffer size must be as returned by fLmdConnectMbs.

uint32_t *Elements

Returns number of elements in buffer.

uint32_t *Bytes

Returns number of bytes used in buffer.

Returns

LMD__SUCCESS
LMD__TIMEOUT
LMD__FAILURE
Mostly if MBS has shut down.

Description

User buffer is filled with data from MBS transport.

uint32_t fLmdCloseMbs(...);

sLmdControl *Control

Returns

LMD__SUCCESS

Description

Close connection, free internal memory.

void fLmdPrintBufferHeader(...);

uint32_t Verbose

1: print, 0: silent

sMbsBufferHeader *BufferHeader

Description

Print buffer header when Verbose is set.

void fLmdPrintFileHeader(...);

uint32_t Verbose

1: print, 0: silent

sMbsFileHeader *FileHeader

Description

Print file header when Verbose is set.

void fLmdPrintHeader(...);

uint32_t Verbose

1: print, 0: silent

sMbsHeader *Header

Description

Print header when Verbose is set.

void fLmdPrintEvent(...);

uint32_t Verbose

1: print, 0: silent

sMbsEventHeader *EventHeader

Description

Print event header when Verbose is set.

void fLmdPrintControl(...);

uint32_t Verbose

1: print, 0: silent

sLmdControl *Control

Description

Print control structure, file header, and current element header when Verbose is set.

Support in f_evt

The standard API f_evt is able to connect to MBS transport in DABC mode or open a file with new format for read. The new format can also be written, by specifying number of streams to 0 in the open call.

-- HansEssel - 30 Oct 2007