RTES Demo System 2004

Overview

The DemoSystem2004 project was/is an exercise to highlight and test the salient features of the RTES project: system specification and modeling, execution oversight with process protection, and low level fault detection with hierarchical mitigation. The demonstration system runs on 80+ dual-CPU Linux nodes, and emulates the operation of the (reduced) L2/3 trigger farm for BTeV.

The DemoSystem2004 consists of:

• GME (generic modeling environment) graphical models for the system integration level, messaging types, run control state machines, user interface definition, and custom ARMOR elements (below), as well as metamodels which define the domain-specific graphical languages for each of the above.
• ARMORs (adaptive reconfigurable mobile objects for reliability) which provide intrinsically reliable high level process oversight, with the ability to stop/restart applications in-place (on the same node), or to relocate the work to an available alternative node. ARMORs insure that protected processes are always running, somewhere, and ARMORs protect themselves as protected processes. ARMORs are modular, and support custom-developed "elements" which can have application-specific fault sensitivities and mitigation behaviors beyond the simple stop/restart.
• VLAs (very lightweight agents) which provide low impact, low level detection of fault conditions, and via Elvin communication can collaborate with other VLAs and ARMOR custom elements to effect local, regional, and global responses to fault conditions.
• physics data source application (GEANT-data file reader), event builder application, trigger application, TCP data channel (source to workers), Elvin command/control channel (all nodes, to/from local, regional, and global control), as well as Ganglia and Matlab-based GUI infrastructure.

DemoSystem2004 was formally presented for critical review at the High Performance Fault Adaptive Large Scale Embedded Real-Time Systems (FALSE-II) workshop associated with the Real-Time and Embedded Technology and Application Symposium (RTAS 2005), 7 March 2005, in San Francisco. Additional information and publications are available as a result.