XDAQ run environment classes for DABC

We use the XDAQ framework for the general set up and controls of the DABC applications. Natively, XDAQ provides configuration of applications by xml files and controls via http and SOAP commands. Control parameters are kept by name in infospace collections. Each XDAQ application is ruled by a finite state machine; the user may implement individual state transition functions.

Besides these features, DIM was chosen as controls protocol for DABC. This required to develop an adapter architecture between XDAQ infospace and state machines, and the DIM services and commands.

We use namespace dabc::xd for the new classes. Location in project: $DABCSYS/dabc/xdaq/dabc/nodecontrol.

View Class diagram

XDAQ runtime and general controls

dabc::xd::Application

The basic XDAQ application class, defining the finite state machine and the web interface. All controllable applications inherit from this class.

• subclass of xdaq::WebApplication with asynchrounous finite state machine fsm. Additionally, there is web state machine wsm as hyperdaq default home page
• state machines implement regular states as defined in the dabc core: Halted, Configured, Ready, Running. Additionally: Error state to be entered if an error happens in any state, and Failure to be entered if a failure occurs during a state transition.
• For the wsm, transition functions are defined : Configure(), Enable(), Start(), Stop() , Halt(), Error() , These implicitely call corresponding transition funtions of fsm: ConfigureAction(), EnableAction(), StartAction(), StopAction(), HaltAction() , ErrorAction(). If error exception is thrown within one of these transition functions, method FailureAction() is executed before putting the state machine to Failure. The fsm may also be switched by appropriate SOAP or DIM commands.
• User API: Transition functions of both the fsm and the wsm call virtual functions DoConfigure() , DoEnable() , DoHalt() , DoStart() , DoStop(), DoError() . These are to be implemented in subclasses. Accordingly, method DoHandleFailure() gives possibility to react on failure case automatically in subclass.
• After reaching the target state, method AfterStateChangeAction() is executed for regular states, and ChangedToFailure() in failure case. Here wsm, fsm, and DIM state records are synchronized.
• method actionPerformed() reacts on item change event and calls Registry::UpdateDIMService() for the item name. Additionally, after item change event of both tags "nodeDNS" and "portDNS", it starts the dim server. This happens after the values are read from the xml configuration on startup. So the actual dim server settings can be passed before startup (NOTE that it is a bad idea to start and stop the dim server several times with different parameters! so we start it at this moment only). Furthermore, this method also takes care of resetting some parameters to the buffered runtime values whenever they are changed from outside, e.g. by loading a configuration file.
• User API: method StatusMessage(string,priority) updates message string in infospace/DIM variable statusMessage (by invoking the Registry) and pushes it to the xdaq message logger LOG4CPLUS. Priority is of type dabc::xd::nameParser::recordstat and may be MESSAGE , WARNING , or ERR ( TODO: add debug level! ). Controls GUI may put message in logger window sorted by the update time.
• User API: virtual method UpdateState() is called by framework to refresh the state machine state representation whenever needed. This updates both a serializable string registered under name "State" that contains fsm state name, and a more complex StatusRecord of the Registry which is displayed as coloured button on the GUI.

dabc::xd::Registry

Defines the controls interface and a registry for commands and parameters.

• uses dabc::xd::DimServer singleton as interface to the DIM server. Parameters for server (nodeDNS, portDNS) are exported to infospace and may be set in xml configuration file before starting the server.
• Methods StartDIMServer() and StopDIMServer() to actually initiate and terminate the service. Note that dim service is not started in constructor. Explicit call is required, e.g. during configure. Currently, we start dim server after its parameters were read from the xml file (see below)
• Method RegisterInfoSpaceAndDIM(): xdata serializable variable will both be exported to application infospace and registered as DIM service. Optionally, xdaq listeners for item change and item retrieve events are attached to the infospace variable. This method may replace direct call of getApplicationInfoSpace()->fireItemAvailable(..) as used in examples.
• User API: Method RegisterSerializable(name,ser, changeable) to register serializable variable to infospace and as dim service. Optionally, changeablitiy by control system may be forbidden. Uses internally RegisterInfoSpaceAndDIM() with all item listeners enabled.
• User API: Method AddControlParameter() registers any simple variable (implemented: int, unsigned int, float, double, bool, std::string) by reference to infospace and DIM. Note that these variables need not to be xdata::Serializable, in contrast to RegisterSerializable(). Uses internally RegisterControlParameter() to add new dynamic serializable to dabc::xd::ControlParameter container collection. Method ChangedControlParameter(void)* must be called from user code to update the infospace and DIM representation if the referenced variable has changed. Vice versa, if the infospace/DIM representation is changed by external command, the container listener function ControlParameter::actionPerformed() will update the referenced variable automatically. RemoveControlParameter() removes any of these dynamic variables.
• Method UpdateDIMService(name) to tell dim server to update service of name. Depending on dim service type, the contents of xdata strings are copied to dim registered character array here. This method should be called whenever xdaq item change listener signals that this item has changed. All registered dim services are kept in a std::vector of pointers to the container object dabc::xd::DimServiceEntry
• User API: Method DefineSOAPCommand(std::string name) registers new SOAP command by name.
  • virtual method OnSOAPCommand(name, parameter, envelope, msg) is invoked by SOAP callback HandleSOAPMessage() .
  • Most SOAP commands are put into a local command queue using PushLocalCommandQueue() and executed asynchronously by a generic command thread which calls the virtual dabc::xd::Application::ExecuteLocalCommand(com,par) .
  • All SOAP commands have a name (first body element name). Command element may have a string parameter as child element.
  • Return message can be put into xoap::SOAPEnvelope reference. Optionally, the OnSOAPCommand() has access to full SOAP message for more complex command objects.
  • Predefined SOAP state machine commands: Configure, Enable, Halt, Start, Stop. Note that Halt is also used to change from FailureState to Halted.
  • Other predefined SOAP commands:
    • SetParameter, to change value of infospace/DIM variable
    • GetInfospace, stream complete infospace variables into SOAP response message (handled in OnSOAPCommand())
    • GetStatus, get state machine string in SOAP response (handled in OnSOAPCommand())
• User API: Method SendSOAP(cmd, par, node, targetid) allows to send SOAP command cmd with string parameter par to an xdaq application on context node with targetid.
• User API: Method DefineDIMCommand(name) : To setup simple (char array) DIM commands, e.g. to change the fsm state. Command names for DIM and SOAP may be (mostly) the same, since both commands are executed in a common method Application::ExecuteLocalCommand, unless some SOAP commands require special response message.
  • All defined DIM commands are kept in a std::vector of pointers to DimCommand objects.
  • virtual method OnDIMCommand(..) : is called by HandleDIMCommand() from dabc::xd::DimServer whenever a command is scheduled from a (controlling gui) client.
  • All received DIM commands are put into a local command queue using PushLocalCommandQueue() and executed asynchronously by a generic command thread which calls the virtual dabc::xd::Application::ExecuteLocalCommand(com,par) . The default will evaluate possible state machine changes by command name. More command actions can be implemented by overwriting this method in application subclass.
  • Predefined DIM state machine commands: Configure, Enable, Halt, Start, Stop. Note that Halt is also used to change from FailureState to Halted.
  • Other predefined DIM commands: SetParameter, to change value of infospace/DIM variable
• User API: DIM client functionality: methods SendDimCommand() and SendDimCommandAsync() to propagate dim command to another dabc xdaq dim server. Note that xdaq application descriptor is used here to address the DIM command, i.e. the real DIM path name is transparent to the caller. The asynchronous variant uses an output queue for the dim commands and sends in dedicated thread; it turned out that synchronous send of dim commands causes troubles when this command is a dabc core reply to a dim command that was just received (case of socket connect commands). i.e. the send operation would actually run in the dim receiver thread, and dim refuses to do that (?).
• User API: virtual method OnErrorDIMServer(..) : is called by dabc::xd::DimServer whenever a DIM error occurs. Default action is to forward message to xdaq LOG4CPLUS error logger. May be redefined in subclass.
• User API: virtual method OnExitDIMServer(..) : is called by dabc::xd::DimServer when any client sends EXIT command. By default, server process termination is suppressed. May be redefined in subclass to allow specific client to shutdown the xdaq process.
• User API: virtual method OnExitDIMClient(..) : is called by dabc::xd::DimServer when client exits. Default operation is debug output and nop. May be redefined in sublcass to react on termination of specific client
• helper methods BuildDIMName() and ReduceDIMName() to transform local infospace name into DIM name that should be unique in the whole DIM namespace (scope of the dim name server). Currently, we use prefix of xdaq context name (url), application class name, and local id number. Used by RegisterInfoSpaceAndDIM() , DefineDIMCommand(name) , HandleDIMCommand(..) , etc.
• User API: Method AddRateRecord() , AddStatusRecord() , AddHistogramRecord() , AddSimpleRecord() will register control record variables for ratemeter, status information, histogram data, and simple types, respectively, to infospace and DIM services. Corresponding Update...Record() methods will refresh data from external reference. RemoveRecord(name) deletes the record entry by name.
• User API: Method ClearHistogramRecord(name) will reset the datafield of histogram record; method FillHistogramRecord(name,bin,value) fills value (default=1) into data array at position bin.
• Method RegisterModuleParameter() is implicitely used by the Add...Record() methods to register ControlParameter subclass ModuleParameter; this class has a reference to the dabc::Parameter. When registered serializable parameter is changed, the value of the parameter reference in the module is synchronized by invoking a change command on the Parameter.
• Method RegisterModuleCommand() is used to export a command from the module manager to dim. Command properties are published as special DIM service by implicit call of AddCommandDescriptorRecord(). Method UnRegisterModuleCommand() removes command services.
• User API: Methods SubscribeParameter() (and UnsubscribeParameter() , resp. ) will subscribe (unsubscribe, resp.) a dabc::Parameter to be updated whenever a corresponding remote parameter in the daq cluster is changed. This feature is implemented by means of ::DimService update mechanism. Subscriptions are kept as vector of dabc::xd::DimParameterInfo (subclass of ::DimStampedInfo with back reference to the subscribed dabc::Parameter)

dabc::xd::DimServer

Subclass of ::DimServer, implementing command handler, error handler, and exit handlers for client and server exit events. Because most DIM server actions are invoked by static methods of ::DimServer, it is reasonable to have only one instance of dabc::xd::DimServer: We design this class as singleton pattern.

• Access and initial creation of the dim server provided by method dabc::xd::DimServer::Instance(). A safe cleanup is granted by dabc::xd::DimServer::Delete() (ctors and dtors are private and cannot be invoked directly).
• By means of method RegisterApplication() each dabc::xd::Registry registers itself in its constructor. Thus the dabc::xd::DimServer has backpointers to all dabc::xd::Registry s in the XDAQ executive and uses actually callback methods of this class as handler functions. Subclasses of dabc::xd::Application may override the virtual command handling method ExecuteLocalCommand() without touching dabc::xd::DimServer class. Messages and exit events are forwarded to all registered applications, commands only to the Registry that matches with its application class name and local id number (prefixes in dim names for services and commands to ensure unique naming).
• Thread safety is taken care of by mutexes (class BSem of xdaq): a static globalMutex to protect the instance pointer, and a appMutex for the application registry.

dabc::xd::DimServiceEntry

Container to keep DimService reference together with backpointer to xdata::serializable variable in infospace.

• Method InitService() creates appropriate DIM service for a given serializable of any known type.
• Method UpdateBuffer() synchronizes DIM service if infospace variable changes (provided by item change event mechanism). Optionally a char* array is used as a buffer here. This buffer is used for updating structure- like serializable types: xdata::string, xdata::Bag(dabc::xd::RateRecord), etc. Simple serializable types expose their data fields without buffering to the DIM service.
• Method SetValue() sets any changeable infospace serializable to a new value, as defined by a string expression.

Interface to DABC core framework

dabc::xd::Node

The application class for all dataflow scenarios, subclass of dabc::xd::Application

• aggregates a dabc::xd::Manager as interface to the core objects.
• Method InitManager() is called in constructor to build a dabc::xd::Manager and invoke a global function InitUserPlugin() that is implemented by the user code to create the dabc::ApplicationPlugins .
• Implements state machine transitions DoConfigure(), DoEnable(), etc. These use Manager::DoStateTransition() to forward transition commands to the core system and the ApplicationPlugins .
• Method InvokeStateTransition(name, Command) to switch a transition of the XDAQ fsm from the core framework. Command in argument is used later to notify completion of state transition back
• Method UpdateState() is re-implemented to notify state change commands from the Manager when state transition is completed. Calls NotifyStateChangeDone().
• Methods SendManagerCommand() is defined here as interface for the dabc::xd::Manager to to send remote dabc manager commands. Overwritten method ExecuteLocalCommand() calling ExecuteManagerCommand() will provide DIM as a transport layer for such manager commands, using the DIM client/server functionality of base class dabc:xd:Application .
• Methods FindManagerName() , FindNodeID() , GetLocalID() , NumberOfAllNodes() to pass information of the cluster setup via the manager to the user implementation classes of the core.

dabc::xd::Manager

Subclass of dabc::Manager, the object manager of the core.

• Implements SendOverCommandChannel() to transport core commands over the dabc::xd DIM layer.
• Implements ParameterEvent(): Interface method to register and update dabc parameter to infospace and DIM. If parameter is created, its contstructor calls this method. Depending on parameter type it is registered using Add...Record() methods of Registry. Vice versa, on parameter deletion it is removed via RemoveRecord(). If module parameter changes value, this is forwarded using Update...Record().
• Implements CommandRegistration() : Interface method to register or unregister a module command as a DIM command and corresponding command descriptor service. Uses internally dabc::xd::Registry::RegisterModuleCommand() and ...UnRegisterModuleCommand() methods.
• Implements ModuleException() to react to a dabc::exception if thrown in a module. So far, we only do error output here. TODO: concept of exception handling in modules, threads and control system!
• Implements methods for the daq cluster management: GetNodeName() , NodeId() to map the unique node id number as used in the core with an arbitrary node name of the configuration/control system; NodeId() delivers the id of the current node. NumNodes() returns the number of all configured cluster nodes as known from the set up. These methods are just forwards to corresponding interface methods of Node.
• Implements Subscribe() and Unsubscribe() for parameters. These methods are just forwards to corresponding interface methods Registry::SubscribeParameter() , and ..:UnsubscribeParameter() , resp.
• Implements InvokeStateTransition(). This method is forwarded to corresponding interface method of Node.

Special node applications

Bnet functionality is achieved by a generic implementation of some xdaq application classes. The goal is that the user set up is completely done by dabc core plugins and the xml configuration files. There is no need to implement and compile further subclasses in the xdaq environment.

dabc::xd::Controller

The controller application that is used to send the connection commands to the worker nodes; subclass of dabc::xd::Node. Sets Manager into the "Main Manager" mode.

dabc::xd::Worker

The worker application that actually works on the dataflow; subclass of dabc::xd::Node. Sets Manager into the "NOT Main Manager" mode.

dabc::xd::Readout

Subclass of dabc::xd::Worker without any further functionality except for the class name that shall describe the purpose on the controls gui.

dabc::xd::Eventbuilder

Subclass of dabc::xd::Worker without any further functionality except for the class name that shall describe the purpose on the controls gui.

-- JoernAdamczewski - 07 May 2008

Scripts for XDAQ environment

Generation of xml configuration file from templates and list of nodes

For testing on multiple node cluster, a shell script is used to extract from a "configuration file template" a cluster configuration xml script for all nodes. Location in project is at dabc/xdaq/dabc/nodecontrol/xml.

preparesetup.sh

Contains xml header and footer string. Defines class name variables for worker (WORKER) and controller (CONTROLLER). The initial values of the instance numbers (INSTCOUNTA, INSTCOUNTB) are set and incremented for each node. Requires two xml template files and a nodelist text file as input.

• template examples DABCBnetExampleWorkerTemp.xml and DABCBnetExampleControllerTemp.xml describes setup for the "B-net scenario". Port numbers of the contexts are 1968 for the controller, and 1972 for each worker node, resp. Variables XDAQNODE, WORKERCLASS, CONTROLLERCLASS, WORKERINSTANCE, CONTROLLERINSTANCE , and CONTROLLERNODE are replaced by preparesetup.sh script.
• Example of node list: nodelist.txt. Expects one node per line; read from preparesetup.sh. Instead of nodelist file, it is also possible to pass nodenames as separate command line parameters (up to 8 nodes supported then).

Usage: ". preparesetup.sh DABCBnetExampleControllerTemp.xml DABCBnetExampleWorkerTemp.xml nodelist.txt"

or

". preparesetup.sh DABCBnetExampleControllerTemp.xml DABCBnetExampleWorkerTemp.xml lxi003 lxi004 lxi005"

This will produce the real setup file DABCBnetExampleWorkerSetup.xml (i.e. the "Temp" in node template file name is just replaced by "Setup").

preparesetup_ib.sh

For the infiniband cluster depcp001.gsi.de, there is a different set of files working in the same way: preparesetup_ib.sh, DABCBnetClusterWorkerTemp.xml, DABCBnetClusterControllerTemp.xml, and nodelist_ib.txt. NOTE: Here the names of the output files are fixed to DABC_IB_Setup.xml .

preparesetup_mbs.sh

For setups using the mbs libraries, there is a different script to work in the same way: preparesetup_mbs.sh, using DABCBnetMbsReadoutTemp.xml, DABCBnetMbsEventbuilderTemp.xml, and==DABCBnetExampleControllerTemp.xml, and nodelist_evbuild.txt. NOTE: Here the names of the output files are fixed to DABC_MBS_Setup.xml .

DABC_ABB_Setup.xml

Not a template, but a complete example configuration file to run the bnet test on the IB cluster together with the ABB board.

Starting XDAQ executives on cluster

All startup/login scripts are at $DABCSYS/dabc/scripts.

dabcxdaqinit.sh

A script for fast startup after login this will set the XDAQ, dabc and DIM environments. Libraries are expected to be in $DABC_LIBS, configurations in $DABC_CONFIGS.

copyfromdeveldabc.sh

Will copy the appropriate libs from $DABCSYS (developer workspace) to these installation locations (important if developer workspace is not visible on remote nodes).

startxdaq.sh

To start xdaq executive with parameters:

Usage: . startxdaq.sh PORT CONFIGFILE DABC_CONFIGS DIM_DNS_NODE

e.g. . startxdaq.sh 1972 dabc_IB_setup.xml /misc/adamczew/xdaq_configs lxg0517.gsi.de

Sets current directory to $DABC_CONFIGS, where it expects XDAQ configuration file $CONFIGFILE . The DIM server must run on $DIM_DNS_NODE. This will start a XDAQ executive $HOSTNAME:$PORT on the local machine. Standard output is written to file $DABC_CONFIGS/dabclog-$HOSTNAME-$PORT .

dabcstartup.sh

Script that actually is invoked from the (Java) GUI to launch the executives on all nodes of the cluster via ssh. Uses startxdaq.sh on each node with parameters taken from GUI setup window. List of nodes is taken from xdaqnodes.txt in working directory DABC_USER_DIR.

Usage: dabcstartup.sh DABCSYS DABC_CONFIGS CONFIGFILE DIM_DNS_NODE DABC_MASTER_NODE

Parameter $DABC_MASTER_NODE is node to run the master state machine / cluster controller. Other parameters are same as described above.

killxdaq.sh

To kill xdaq process on current node. Uses ps -x to figure out all processes that match given name substring

Usage: . killxdaq.sh PROCESSNAME

e.g. . killxdaq.sh xdaq

dabcshutdown.sh

Script that actually is invoked from the (Java) GUI to kill the executives on all nodes of the cluster via ssh. Uses killxdaq.sh on each node. List of nodes is taken from xdaqnodes.txt in working directory DABC_USER_DIR.

-- JoernAdamczewski - 24 Apr 2008