Next: III. User Manual: FlowVR Up: FlowVR : A Middleware Previous: I. Getting Started Contents
The FlowVR library provides a programming and execution environment for interactive distributed applications. Its goal is to enforce application modularity for leveraging software engineering issues while enabling high performance executions on parallel and distributed architectures. FlowVR first targets executions on multi-processor machines, PC clusters and Grids.
FlowVR is based on a hierarchical data-flow model. A FlowVR application is made of a collection of metamodules that spawn modules running on different machines. Modules are assembled through a network that enables data exchanges.
To get a good understanding of the FlowVR concepts, read the publications available at http://flowvr.sourceforge.net/FlowVRPublications.html.
A module is a single thread of execution running a potentially infinite loop. At each iteration a module gets data from input ports and sends results on output ports.
A Metamodule is specific component that associates a launching command to a set of modules. A code running just one module is the simplest metamodule. A parallel or multi-threaded code (a MPI parallel code or C++ code using POSIX threads for instance) can run several modules on one or more machines. Such codes are generally launched from a single command (mpirun ... for instance). In this case we usually associate them to the same metamodule.
Data transfered between modules are encapsulated into messages. A message has an arbitrary size. Each message is associated with a list of stamps, a lightweight data used to route or filter messages. This list of stamps can be duplicated and routed separately from its message to special network nodes in charge of synchronization policies. We call such a meessage a STAMP message in opposite to a FULL message. Besides predefined FlowVR stamps, others, like a time or a 3D bounding box for instance, may be added to extend the message handling operations that can perform the network.
For message exchange, the input and output ports of modules are connected through a network. This network is composed of several components:
These components enable to design complex networks including collective communications, synchronization barriers, resampling patterns, dynamics routing based on bounding boxes for instance, etc.
|
Programming a FlowVR application is a two step process:
It is strongly adviced to keep components independent from the target architecture. FlowVR comes with a mechanism to read architecture specific data from external files to automatically build an instance of the application adapted to the target architecture.
Developers can create their own filters and synchronizers if the ones provided with FlowVR do not suit their needs.
The FlowVR runtime engine relies on daemons, one per participating node. All components of the application network and the message exchange between modules are managed by daemons. Daemons act as a brokers and relay messages between modules. Filters, routing nodes and synchronizers are implemented as dynamically loaded classes ( plugins) within the daemon.
Each daemon manages its own shared memory segment. To prepare a new message, a module requests to the daemon a free buffer in the shared memory. Once the message is ready, the module warns the daemon. Next, the daemon consults its internal data to identify what should be done with this message. If the message must be sent to a distant node, the daemon will take care of it: it will send the message to the daemon of this distant node.
When a module requests a new message, the daemon first needs to identify the message to be delivered. Once this message available in the shared memory segment, the daemon returns to the module a pointer to the buffer where the message is stored. This approach enables to implement all the data exchange logic at the daemon level. It has two main advantages. It eases the metamodule programming by making modules "network blind". It also enable optimizations like avoiding unnecessary message copies by giving modules direct access to the shared memory.
The basic implementation of inter-daemons communications relies on TCP. It ensures a wide portability and enables to easily deploy a FlowVR application on heterogeneous machines. The connections are automatically and dynamically created. Daemons can be launched independently.
The control of the execution of a FlowVR application is managed by one special module called a controller, automatically loaded when launching the application. The controller is in charge of launching the metamodules and setting the network. The controller first starts the application's metamodules. Once the modules launched, they register to their local daemon that sends an acknowledgment to the controller. Then, the controller sends to each daemon the list of plugins to load to implement the FlowVR network.
Here is the list of source codes a user may find useful to inspect:
A code documentation generated by Doxygen is available for some packages. It is available online. You can also generate it locally, executing:
make doc
Throughout this chapter we use as a running example a distributed application that iteratively computes prime numbers and displays them over a spiral. The user sees the pattern growing while the prime numbers are computed. He can interactively rotate the view (share/flowvr/examples/primes). We distinguish three parts :
|
We advice to use the share/flowvr/examples/primes example as an application template to start developing your own application.
All the the examples from share/flowvr/examples follow the same organization: This directory contains:
All provided examples fellow the same organisation based on Cmake for the compilation.
We here just quickly list the different steps for a local execution of the Primes example. Refer to TicTac compilation and exeuction instructions for more details.
Compile (simple script calling cmake for a local installation):
./make-app.sh
Launch a daemon:
flowvrd -top
Load the configuration file:
. bin/primes-config.sh
. bin/primes-config.csh
Start the application on you local host:
flowvr --localhost --launch Primes
You can use the arrow keys for some interaction on the visualization, but be sure to have the right window focused (the small one whith the instruction text written to). To stop the application (not the daemon), enter:
stop
If something goes wrong, kill zombie processes using:
flowvr-kill
We detail here the content of the primes-config.sh (the primes-config.csh content is similar but for a csh shell):
share/flowvr/examples/primes/bin/primes-config.sh
Modules and network components send and receive messages. This section describes how to handle these messages. It is important to underline that what is presented is relevant for module programming as well as for filter and synchronizer programming.
As FlowVR is optimized for low-latency, all data is exchanged through a shared memory area to minimize recopy. To manage each piece of information we define flowvr::Buffer and the associated classes.
A buffer is a reference to a chunk in the shared memory. The flowvr::Buffer class holds a read-only reference while the flowvr::BufferWrite class contains a writable reference. These classes are very simple and lightweight, all the shared memory management is deferred to an abstract flowvr::BufferImp class.
A new buffer can be obtained by calling an allocation method (see
![[*]](crossref.png)
). This returns a writable reference. The data
can then be written in the buffer by calling either the
BufferWrite::writeAccess() method returning a pointer to
the beginning of the buffer, or the
BufferWrite::getWrite<Type>(offset) method returning a
pointer of the given type to the specified offset in the buffer
(measured in bytes).
When a buffer is received or has been sent the data can only be read. It is possible to read its content through the Buffer::readAccess() method returning a read-only pointer to the beginning of the buffer, or the Buffer::getRead<Type>(offset) method returning a read-only pointer of the given type to the specified offset in the buffer.
When you copy a Buffer only the reference is copied, and a reference counter is updated. You can obtain a reference to a subset of a given Buffer by specifying an offset and size as parameters to the constructor. When a Buffer or BufferWrite object is destroyed the reference counter is decremented, and the associated buffer is automatically freed if no other reference exists.
The following code allocates and fills a buffer to send a set of identifiers indicating currently pressed keyboard keys:
flowvr::MessageWrite msgWrite; // MessageWrite object unsigned char *pMsgData = 0; // Pointer to allocated memory unsigned int keyPressedCount = ...; // Hold the number of keys currently pressed// Request a FlowVR buffer large enough to store pressed keys identifiers : msgWrite.data = pFlowVRModule->alloc(keyPressedCount*sizeof(unsigned char));
// Get writing reference : pMsgData = (unsigned char *)msgWrite.data.getWrite<unsigned char>();
// Fill buffer with the identifiers of the keys which are pressed : for (int i=0; i<MAX_KEYS; i++) if (tKeysState[i]) *pMsgData = (unsigned char)i; pMsgData++;
And below is the cod to read keys state information from a received message:
flowvr::Message msgRead; // Message object unsigned int keysPressedReceviedCount; // Number of keys identifiers to read unsigned char* pKeysPressed = 0; // Application buffer to store keys pressed// Count the number of pressed keys from message size (returned in bytes) : keysPressedReceviedCount = msgRead.data.getSize() / sizeof(unsigned char);
// Copy message buffer to application buffer : if (keysPressedReceviedCount > 0) pKeysPressed = new unsigned char [keysPressedReceviedCount]; memcpy((void*)pKeysPressed, msgRead.data.readAccess(), keysPressedReceviedCount);
Remark The actual codes of capture ((share/flowvr/examples/primes/src/capture.cpp) and visu ( share/flowvr/examples/primes/src/visu.cpp)) use a higher level layer for handling key messages: Chunk Events.
Remark: You can send empty messages simply by allocating a buffer of size zero.
As the shared memory is concurrently accessed by several processes, allocation operations can become quite expensive (lock contentions, memory fragmentation). To remove this potential bottleneck, a simple class lets you reuse old buffer not used anymore. This only works for the case of repetitive allocations of buffers with the same size, which is quite common.
The flowvr::BufferPool let you allocate a buffer similarly to the standard allocation scheme. But behind the scene it stores a cache of old buffers. The maximal size of this cache is specified at the construction of BufferPool. At each allocation request, if the requested size did not change it searches for an old buffer not in use anymore (i.e. whose only reference is this BufferPool) and reuses it instead of allocating a new one.
In our example, the compute component send calculated prime numbers by packets of constant size. Thus buffer pools are used in share/flowvr/examples/primes/src/compute.cpp) :
flowvr::BufferPool* pOutPool = 0; // BufferPool object unsigned int tempPrimeNumbersMaxCount = ...; // Constant count of prime numbers unsigned int *tTempPrimeNumbers = 0; // Calculated prime numbers by iteration// Create a pool of buffers : pOutPool = new flowvr::BufferPool();
...
flowvr::MessageWrite msgWrite; // MessageWrite object
// Request for a new buffer from the pool to send new computed prime numbers. msgWrite.data = pOutPool->alloc(pFlowVRModule, tempPrimeNumbersMaxCount*sizeof(unsigned int));
// Fill message data : memcpy((void*)msgWrite.data.writeAccess(), (void*)tTempPrimeNumbers, tempPrimeNumbersMaxCount*sizeof(unsigned int));
This mechanism of buffer pools offers the benefit of reducing or even removing dynamic allocations with a bounded memory cost (the size of each buffer times the maximal size of the cache).
In particular, if the size of buffers is not constant but bounded, a possibility is to allocate a pool of buffers of maximum size. When a message should be written, such buffer is requested and another buffer of necessary length is derived from it by copy. Let's recall that multiple copies of a buffer are simply references that share the same memory area.
The code sample below shows how to derive a buffer of size 256 from an original buffer of greater size :
BufferWrite originalBuffer; // Buffer whose size will be >= 256... // Deal with originalBuffer
BufferWrite* pSubBuffer = 0; // pointer to the sub-buffer
// Get a reference to a subset of the original buffer (the 256 bytes at offset 0) : pSubBuffer = new BufferWrite(originalBuffer, 0, 256);
We define a flow of messages as the ordered set of messages sent through a specific output port of a given module (this module is called the source of the flow).
A stamp is a small piece of information related to the data of the message. Stamps are used to identify the messages and processes as required by the application.
Each stamp has a name and a type. Currently the following types are supported:
Several stamps are automatically defined by FlowVR:
These stamps are used to provide the logic of the connections between modules. For example, when we are reading a new message, we can use its source stamp to identify the output port this source is related to and send it to the appropriate node. If message exchange fellows a FIFO mode, each newly received message will have its num stamp equal to the previously received message's num plus one.
Remark: The reader can skip this section if he does not need user defined stamps.
Stamps are exposed through the following classes:
During the initialization phase, each FlowVR module specifies the list of stamps of its ports. This is achieved by adding stamps definition (StampInfo) to the StampList of each port. A StampInfo is constructed by specifying the stamp's name and type. This type is created by calling the create method of the appropriate Type subclass. A stamp is then added to the StampList using the method add.
As an example, our computing modules compute informs the visualization module visu about the time that was necessary to process the prime numbers sent at the last iteration. This duration in microseconds simply comes with the primes message in the form of a stamp named computationTimeIt.
On the sender side, compute, the definition of this user stamp is appended to the output port object dedicated to primes numbers (see share/flowvr/examples/primes/src/compute.cpp) :
// Output port declaration :
flowvr::OutputPort* pPortPrimesOut = new flowvr::OutputPort("primesOut");
// Stamp declaration :
flowvr::StampInfo *pStampComputeTime = new flowvr::StampInfo("computationTimeIt", flowvr::TypeInt::create());
// Associate stamp with output port :
pPortPrimesOut->stamps->add(pStampComputeTime);
And on the destination side, visu, the stamp must be declared in a similar way (see share/flowvr/examples/primes/src/visu.cpp) :
// Input port declaration :
flowvr::InputPort* pPortPrimesIn = new flowvr::InputPort("primesIn");
// Stamp declaration :
flowvr::StampInfo *pStampComputeTime = new flowvr::StampInfo("computationTimeIt", flowvr::TypeInt::create());
// Associate stamp with input port :
pPortPrimesIn->stamps->add(pStampComputeTime);
To write the values of a stamp in a new message you must call the write method of the StampsWrite class. This method needs as arguments the description of the stamp to write and its new value. In our example, inside compute, this leads to:
flowvr::MessageWrite msgWrite; int lastIterationComputeTime; // .... // msgWrite.stamps.write(*pStampComputeTime, lastIterationComputeTime);
Similarly, a stamp is read using the read method. Note that this method returns true if the operation succeeded and false otherwise. In the case where the stamp is not present in the received message, an error is returned when the module reads the stamp. Then you can either decide to ignore the error and use a default value instead, or stop the execution of the module and warn the user. Our visu example module sets computation time to -1 (indicating no time was provided) if the stamp is not present. The code to read the stamp N in this case is:
flowvr::Message msgRead;
int lastIterationComputeTime;
// .... //
if (!msgRead.stamps.read(*pStampComputeTime, lastIterationComputeTime))
lastIterationComputeTime = -1;
As stated before, FlowVR provides some predefined stamps on each connection, in particular the message iteration counter it. The visu module makes use of this stamp two times in order to display a comparison in the iteration rate for each of the three modules. As the stamp is predefined, the stamp can directly be read over the port without declaration (see share/flowvr/examples/primes/src/visu.cpp) :
int iterationCounterComputes = 0; int iterationCounterCapture = 0;// Read predefined 'it' stamp in order to know the iteration state of compute node(s) : msgRead.stamps.read(pPortPrimesIn->stamps->it, iterationCounterComputes);
// Read predefined 'it' stamp in order to know the iteration state of capture node : msgRead.stamps.read(pPortKeysIn->stamps->it, iterationCounterCapture);
Remark: Accessing the cells of an array of stamps is done by appending the cell index inside [] after the StampInfo instance.
A Module is a single thread of execution running a potentially infinite loop. At each iteration a module gets data from input ports and sends results on output ports. The module API has 5 main instructions: init(), wait(), get(), put(), close(). The module API is not thread-safe, i.e. if several execution threads share the same module, the programmer must ensure a proper thread synchronization. Also note that several modules can be interlaced into a single thread of execution.
A Metamodule is specific component that associates a launching command to a set of modules. A code running just one module is the simplest metamodule. A parallel or multi-threaded code (a MPI parallel code or C++ code using POSIX threads for instance) can run several modules on one or more machines. Such codes are generally launched from a single command (mpirun ... for instance). In this case we usually associate them to the same metamodule.
In the following we detail the syntax and semantics of the FlowVR Module API based on a C++ syntax. This API is simple to limit the work required to program a module or turn an existing code into a module.
Each module has two predefined ports. The beginIt input port and the endIt output port:
The Primes example heavily uses endIt and beginIt ports for dataflow synchronization purpose.
Refer to share/flowvr/examples/primes/src/compute.cpp to see the complete code of a (meta)module. For a simpler code example, you can refer to the tic-tac example.
To alleviate the module programming task, FlowVR offers a high-level function for registering and initializing a module. This function is a simple alternative to the manual module creation task.
static ModuleAPI* initModule(std::vector<Port*>& ports, std::string instancename=std::string(""));
This method first registers the module to create the appropriate module API implementation. The module's name is composed of the concatenation, with a "/" separator, of those fields:
Notice that in most cases the instancename parameter can be omitted. After being registered, the module and its ports are initialized. If an error occurs during these steps an error message is emitted and NULL is returned. Here is an example on how the compute module is simply created with the following code (extracted from share/flowvr/examples/primes/src/compute.cpp)
// Registers and initializes the module to the FlowVR daemon :
if (!(pFlowVRModule = flowvr::initModule(ports)))
return -1;
int ModuleAPI::wait()
Notice that the wait call will wait for the availability of incoming message only on the input ports attached to a connection.
int ModuleAPI::get(InputPort* port, Message& message)
Notice that during an iteration (i.e. between two wait()), calling get() on a given port will always return the same message.
int ModuleAPI::put(OutputPort* port, MessageWrite& message)
Sending a message before the first call to wait let a module send a first message without waiting for any input (this can be considered as a default value message). This is mandatory if the module can be involved in connections cycles (actually this should be done for all modules to ensure they can be reused in any context). In an application where there is a cycle, at least one module must send a message without waiting any input. If this is not the case the application will deadlock as no module will be able to start.
Notice that the message should not be modified after calling this method.
int ModuleAPI::getStatus()
This method return the status of the module, which is zero if an error occurred. All other methods also return the status after completion.
int ModuleAPI::close()
This method should be called when the application is stopped.
BufferWrite ModuleAPI::alloc(int size)
This method allows the developer to allocate a new buffer with a specified size. This buffer can be used to store the data of the messages before being transmitted
Note that, as all other ModuleAPI methods, this method should not be called before init(). This is an important requirement as sometimes allocations can be used to construct some data structures. They might be incorrectly initialized if this is done before the call to the init method.
Dynamic allocations can be costly and memory fragmentation issues can
appear if too many allocations are requested. When a module often
allocates buffers of the same size, it is more efficient to take
advantage of the
flowvr::BufferPool
(see section ) to reduce the required dynamic
allocations.
We strongly advice to rely on CMake to handle module compilation. See for instance the share/flowvr/examples/primes/src/CMakeLists.txt file called from the main share/flowvr/examples/primes/CMakeLists.txt file when configuring and compiling the application.
Alternatively, we also provide pkg-config configuration files to provide the compiler with the right FlowVR specific options. With gcc, it should look like:
g++ -o ./bin/compute ./src/compute.cpp `pkg-config flowvr-mod -cflags -libs`
A component must be developped for each module and each metamodule. See the module component related section and the metamodule component one.
We detail in this section how to develop and assemble the various components required to build an application.
Because FlowVR is designed for large applications, the application network, also called the data-flow graph, can be complex. We provide the user tools to face this complexity and avoid him the burden of explicitly describing such a graph. We rely on the composite design pattern to support hierarchies of components. It enables to encapsulate in one component a complex pattern recursively built from simpler ones.
A component defines input and output ports. We distinguish two kinds of components:
A link connects two component ports. It cannot directly cross a component membrane. A link between 2 ports is allowed only in the 2 following cases:
All components files (header and source files) uses the .comp extension. All components provided with FlowVR are stored in include/flowvr/app/component.
Application specific components are stored appart. For instance the primes components are located in share/flowvr/examples/primes/include/primes/components.
The goal of this section is to give the basics and more important concepts about component programming. Refer to the source file or the doxygen documentation for a exhaustive lists of available methods. The basic methods are defined in
The id of the root component of an application is always the name of its class (with the same case).
For instance in the primes application, the id of the root component is "Primes". It includes an instance of the MetaModuleCapture component with id ``capture'' calling share/flowvr/examples/primes/src/primes.comp.cpp:
MetaModuleCapture * capture = addObject<MetaModuleCapture >("capture");
The full id of the capture component is "Primes/capture".
All components inherit from the Component, Primivite and Composite classes defined in include/flowvr/app/core/component.h.
The constructor with only the id as parameter is mandatory. The constructor should only define the part of its interface that does not depend on other external components or data (not conforming with that rule may lead to errors). It is usually limited to call:
For instance the ModuleVisu component of the primes example defines two ports in its constructors: share/flowvr/examples/primes/src/primes.comp.h
The create method is mandatory for each component. It creates an new instance of the caller component ith the same id. See share/flowvr/examples/primes/src/primes.comp.h for instance.
The part of the interface that could not be defined in the constructor and the content of the component must be set in the execute method. The Primes component uses this method to define its content according to the value of the "example" parameter: share/flowvr/examples/primes/src/primes.comp.h
Each component is associated with a set of hosts that will be used to define where the primitive components it contain will be mapped. Most of the time the inherited setHosts does the job correcty. But some compiste compoents need to overwrite this method.
The root component of an application is compiled in a library that is dynamically loaded by the flowvr command to be processed. This imposes some contraints on the form of this root component:
The call to GENCLASS looks like:
GENCLASS(MyClassNameRespectingCase)
See also share/flowvr/examples/primes/src/primes.comp.cpp.
The addObject method enables to add a new component instance into the caller component. It specifies the type of the component through a template and the id of the create component. This method return a pointer to the create component.
The Primes component (share/flowvr/examples/primes/src/primes.comp.cpp) adds the MetaModuleCapture component with id "capture" calling:
MetaModuleCapture * capture = addObject<MetaModuleCapture >("capture");
The addPort method as 4 arguments the 2 lasts being optionals:
A link connects to ports from different components. A link is not a message channel. A link is allowed only between:
The link component takes as argument a pointer on each port to link.
The addObjectandLink method add an new component and link two of its ports in one call. The names of the ports of the newly added components can be ommitted if they are "in" and "out" ports.
Fro instance in share/flowvr/examples/primes/src/primes.comp.cpp, you can find the following code adding a COnnectionStamps componnent. This componet as an input port "in" and an output port "out". We link the input port "in" with the output "out" port of the sMaxFrequency component and the output port "out" with the input "beginIt" port of the capture component using :
addObjectandLink<ConnectionStamps>("beginIt",sMaxFrequency->getPort("out"),"in","out",capture->getPort("beginIt"));
But using the variant of this method we can ommit to mention the "in" and "out" ports and simply write:
addObjectandLink<ConnectionStamps>("beginIt",sMaxFrequency->getPort("out"),capture->getPort("beginIt"));
Component encapsulation hides their content. Consider the case where one component A contains primitive components . Each of these components has one input port linked to the single input port of A. Let consider a second component B that also contains child components . Each of these components also has one output port linked to the single output port of B. if we just link directly A and B we do not provide enought information to the program so that it can infer the pairs that are connected. We provide some methods that enable to specify this without ambiguity and without impairing the encapsulation provided by components.
The getPrimitiveSiblingPorts methods (include/flowvr/app/core/port.h) return the lists of ports of primitive components that are linked to a given port p. These methods only look at the primitive components outside of the component owning p. There is not limit on how deep these primitive components can be embedded into composite components as long as there exists a chain of links leading to them. These methods should not be used in component constructors.
Using these methods the component A, once linked to B, can obtain the components .
For instance the filtermerge filter needs to know the number of primitive input ports it is connected to to define the number of ports of the enclosing filtermergeprimitive component. filtermergeprimitive is the primitive component that merges its inputs to produce one output.
Now that A knowns the distant primitive components, it can link each of its child component using the specific linkAndConstraint method. This method links one component to an other like link but it constrain this link to be associated to a specific distant primitive component.
A component content may be defined according to the existence of other components connected on its ports. We classicaly need to know the list of primitive components connected to a given port. We provide a set of methods that return the lists of ports of primitive components that are connected to a given port p. Notice that these methods only look at the primitive components outside of the component owning p.
Component encapsulation hides their content. Consider the case where one component A contains components . Each of these components has one input port linked to the single input port of A. Let consider a second component B that also contains child components . Each of these components also has one output port linked to the single output port of B. If we just link directly A and B we do not provide enought information to the program so that it can infer the pairs that are connected. We provide some methods that enable to specify this without ambiguity and without impairing the encapsulation provided by components.
An application must be compiled as a shared library so it can be loaded by the flowvr appliction processing command (the root component must have been declared loadable using the GENCLASS macro).
By convention we name this library after the name of the root component (using lower cases only). For instance the Primes aplication is compiled into a library called libprimes.comp.dylib.
We rely on Cmake to compile the library (share/flowvr/examples/primes/CMakeLists.txt and share/flowvr/examples/primes/src/CMakeLists.txt).
For convenience we include the path to the library in the LD_LIBRARY_PATH and DYLD_LIBRARY_PATH environment variables (share/flowvr/examples/primes/bin/primes-config.sh).
The application is processed using the flowvr command. Its main options are:
For instance the primes application is started on the local host using (need a FlowVR deamon running)
flowvr -L -x Primes
To give the example parameter the value 2 and overwrite the default value:
flowvr -L -x -P primes:example=2 Primes
Actually examples 4 and 5 really make sense only if several instances of the compute module are started. Using the -L option prevent to do that. So just remove the -L option tp map the components according to the content of the file (share/flowvr/examples/primes/primes.csv):
flowvr -x -P primes:example=2 Primes
flowvr produces several output files (name matching the application class name):
Application processing is organized around a traversing process. Traversing an application consists in recursively calling a given method on all components (these specific components are called controllers). It is important to understand the traverse process. Each time an exception is called when executing a controller on a component, the state of the component is restored and the traverse calls a controller on a different component. One reason for this rollback is that there is data dependencies between some controller calls. This controller should be called after the controller of a different component. This process iterates up to have all component sucesfully traversed. If the traverse reaches a fix point with list of component all failing to execute the controller, the application processing stops. Two reasons can lead to this failure:
flowvr executes a sequence of 3 traverses:
There are 2 important consequences on component programming:
Parameters are a powerful mechanism to parametrize components making them more generic. A parameter is key/value pair. It is declared by the component that uses it. It is set from this same component, a parent component, or externally from the flowvr command line or configuration files. The value of the component can be retrieved and used by the component that declared it. All parameters of a component are written in the .net.xml file, enabling the associated module or filter to retreive the parameter value when starting.
A component declares its parameter PARAM with a default value "3" (the default value is optional) (include/flowvr/app/core/component.h) calling:
addParameter<int>("PARAM",3)
You can set a parameter's value by different ways:
The setParameter method enables to set the value of a parameter. The call:
setParameter<int>("PARAM",10)
A variant of SetParameter enables to be more precise and set a parameter for a given sub-component and all its sub-components. It requires an extra argument providing the relative id from the caller component. If the compoenent "root/caller" calls:
setParameter<int>("PARAM",10,"relative/path/to/subcomponent")
For instance this variant is used by the primes component (share/flowvr/examples/primes/src/primes.comp.cpp) to selectively set the arity of two different communication trees defined in two different sub-components:
setParameter<int>("TREE_ARITY", 2,"greedyPrimes/ComIn");
setParameter<int>("TREE_ARITY", 4,"greedyPrimes/ComStamps");
Parameters values can be set directly from the flowvr command line.
flowvr -P component/full/id:PARAM=10 [other options] [yourApplication]
To set multiple parameters, use -P option with quotes, separating each parameter with spaces:
flowvr -P "component/full/id:PARAM=10 component/full/id2:PARAM2=20" [other options] [yourApplication]
flowvr -P "component/full/id:PARAM=string value with spaces&component/full/id2:PARAM2=20"
flowvr -Q parameterfile.txt [other options] [yourApplication]
component/full/id:PARAM=10 component/full/id2:PARAM2=string value with spaces ...
If flowvr finds a file names yourapplication.adl.in.xml, it loads it and sets its parameters using the values it contains.
At execution, flowvr dump all parameters used by your application in the yourapplication.adl.out.xml file. This file can be renamed as yourapplication.adl.in.xml and modified as a convenient way to backup a set of parameters. In the adl file each parameter has an attribute "from" specifying the way its value was set.
There are different ways to set parameters and the same parameter can be set several times. For example a parameter can have a default value at declaration, then another value set calling setParameter<type T>() and finally specifying another one with the command line. The actual value of a parameter is set according to the following rules from the highest to the smallest priority (search starts from the component that declared the parameter):
A parameter value can be accessed by a component or at execution
This parameter now can be used as a value that can be "set" and/or "get" by different ways, during execution. It can be useful in many cases : setting a path for a metamodule command line, a number of port for a component ... and all the times you need a value that will be set at execution.
The method
getParameter<type T>("PARAM")
All the parameters used by an application are stored at runtime in yourApplication.adl.out.xml, with debug informations (as the way the parameter as been set : from command line, in the application code...). This file is usually used for debug, or to backup a set of parameters (see above).
Parameters used by filters and synchronisers are also stored in yourApplication.net.xml file. Indeed, filters and synchronisers are handled by the flowvr daemon flowvrd. Each filter or synchroniser corresponds to a plugin, loaded by the daemon. Those plugins read parameters (as trees arity, number of ports...) in yourApplication.net.xml file.
For example, let's have a look on presignal filter, used to generate initial synchronisation messages :
In include/flowvr/app/components/filterpresignal.comp.h, a parameter "nb" is declared, with a default value set to 1. This parameter "nb" can be set, in the application code, to modify the behaviour of the filter. If not, 1 is retained as default value. This value appear in yourApplication.net.xml. At launch time, deamon load corresponding plugin : flowvr-suite/flowvrd/src/plugins/filters/flowvr.plugins.PreSignal.cpp, which read yourApplication.net.xml and set the filter behaviour according to the parameter value written.
For instance in the primes example a parameter example is declared by the root component primes in its constructor, set from the comand line, and used by primes in its execute method to select in between different components assembly schemes.
Parameters are used in example "primes", to define which example to run at launchtime.
A parameter example ith a default value is declared in the constructor of the Primes component (share/flowvr/examples/primes/include/primes/components/primes.comp.h):
addParameter<int>("example",4)
The parameter example is used in the execute method of Primes to select the example to run (share/flowvr/examples/primes/src/primes.comp.cpp):
int example = getParameter<int>("example");
If primes is launched without overiding the value or example, it executes the 4th example because getParameter<int>("example") will return the default value defined at declaration.
If you edit the primes.adl.out.xml, you'll see the following token for the primes component:
<example from="declare">4</example>
The example executed can be changed by specifying a different value through the comand line:
flowvr -x -L -P primes:example=3 Primes
If you edit the primes.adl.out.xml, you'll see the following token for the primes component:
<example from="cmdLine">3</example>
A connection is necessarily point-to-point. Along a connection, messages are not lost and their order is preserved. A connection component has:
FlowVR distinguishes two types of connections:
Both components are defined in include/flowvr/app/components/connection.comp.h.
In practice connections are automatically inserted in the network during the application processing. The user should not have to directly handle connection components.
All filters inherit from the base class include/flowvr/app/components/filter.comp.h.
A filter is instantiated in a network by adding a <filter> node with a unique id attribute and an host attribute specifying the host it will run on. Within this node a <filterclass> node must be added containing the class name of the filter in a java-like notation (full class name with namespace components separated by periods). If the filter requires some initialization parameters they must be specified within a parameters node. Each input port (respectively each output port) of this filter must be declared within a <input> node (respectively <output> node).
Refer to the DTD flowvr-parse/dtd/network.dtd for the exact syntax.
Each time a filter is instanciated, it is added to the .net.xml and .cmd.xml files, indicating to the flowvrd deamon it has to load the corresponding plug-in.
FlowVR provides several predefined filters. Here is a short description for each of them, and the corresponding plug-in :
Discard all messages from input if filter not open, forward them all on output otherwise
Corresponding plug-in loaded by daemon : flowvr::plugins::Discard
Forward only first port (it is a composite not a standard filter)
Send messages at the given frequency (precision unkown)
Corresponding plug-in loaded by daemon : flowvr::plugins::FrequencyFilter
Increment it
Corresponding plug-in loaded by daemon : flowvr::plugins::FilterIncrement
Send 'it' messages on output port, according to 'it' stamp received on input
Corresponding plug-in loaded by daemon : flowvr::plugins::It2Start
Undo filtering performed by 'FilterIt'
Corresponding plug-in loaded by daemon : flowvr::plugins::UnfilterIt
Forward on its output the messages received on its input having the same 'it' numbers than the ones received on the order port. Discard other messages (filter usually used with a synchronizer)
Corresponding plug-in loaded by daemon : flowvr::plugins::FilterIt
Forward on output messages the last input message or null message if there is no input. Discard other messages.
Corresponding plug-in loaded by daemon : flowvr::plugins::FilterLastOrNull
Each time it receives a message on each input port, it sends one message consisting of a combinaison of nb messages receved on inputs in0, ..., innb-1 (combining order given by the mask). Ouput message has stamp of first input message optionally adding the stamp given in parameter.
Corresponding plug-in loaded by daemon : flowvr::plugins::MaskMerge
A shell for FilterMaskRoutingPrimitive. Set the and 'outport' parameter for the MaskRoutingPrimitive filter. MaskRouting Filter should always be used through this component rather than directly with FilterMaskRoutingPrimitive
Corresponding plug-in loaded by daemon : flowvr::plugins::MaskRouting
Split input message in nb parts of size elementsize and send parts to output out0, out1, ..., outNb-1 according to mask
Corresponding plug-in loaded by daemon : flowvr::plugins::MaskScatter
When receiving a signal with stamp i, remove i messages on portIt, and send on output a message with stamp i+nb (nb given as parameter)
Corresponding plug-in loaded by daemon : flowvr::plugins::MaxBuffer
Each time it receives a message on each input port, it sends the maximum value on output port (input data must be an unsigned int)
Corresponding plug-in loaded by daemon : flowvr::plugins::Max (See filterMax)
A shell around a filtermax component with one input port masking the actual number of ports of filtermerge
A shell for FilterMaxPrimitive. Set the 'nb' parameter for the MaxPrimitive filter, giving the number of 'in port'. Max Filter should always be used through this component rather than directly with FilterMaxPrimitive
Each time it receives a message on each input port consisting of a 2D array, it sends one message consisting of the association of the 2 input 2D arrays. It uses the stamps P and N to define how to combine these arrays.
Corresponding plug-in loaded by daemon : flowvr::plugins::Merge2D
A shell around a filtermerge2dprimitive component masking its 2 input ports
Merge messages received on input port into one message sent on output port. More precisely, when it receives a message on the 'order' port, it inspects the icomming message queue ('in' port), discard all messages with a non null 'scratch' stamp value, concatenate all other messages in one message sent on 'out' port. This message has its 'scratch' stamp set to 0 and its 'stamp' stamp set to the sum of all 'stamp' stamps of the concatenated messages. The name of the 'scratch' and 'stamp' stamps are read from the omponent parameter list. These parameters are optionals
Corresponding plug-in loaded by daemon : flowvr::plugins::MergeIt
Merge nb successive messages received and send on output. Messages can be discarded using the stamp given with the scratch parameter. The stamp given with the stamp parameter is combined in the merged message.
Corresponding plug-in loaded by daemon : flowvr::plugins::MergeMsg
Each time it receives a message on each input port, it sends one message consisting of the stamp of the message received on in0 and the concatenation of the date received on in0, in1, ... inNb-1
Corresponding plug-in loaded by daemon : flowvr::plugins::Merge
A shell around a filtermergeprimitive component with one input port masking the actual number of ports of filtermerge
After receiving one new message per input, forward on output the last new message received. The it of this message is changed for the minimum it of the new messages received on inputs.
Corresponding plug-in loaded by daemon : flowvr::plugins::MInit
Forward messages received on inputs in0,...,innb-1 on output. Change the it and num stamps to 'hide' multiplexing
Corresponding plug-in loaded by daemon : flowvr::plugins::MultiplexFilter
Filter that adds nb (a paramater) inital messages, then forward incoming messages to output (used to boot cycles). 1 parameter. 'nb': number of initial messages
Corresponding plug-in loaded by daemon : flowvr::plugins::PreSignal
Corresponding plug-in loaded by daemon : flowvr::plugins::FilterRenum
Alternatively dispatch messages received on in0, in1, ..., inNb-1 on output. 1
Corresponding plug-in loaded by daemon : flowvr::plugins::RotateFilter
A filter for routing messages (full or stamps). Just send each message received on 'in' port on 'out' port. Uses the FilterPreSignal with parameter 'nb' set to 0 (no presignal) by default.
Split input message in nb parts of size elementsize and send each part to output out0, out1, ..., outNb-1
Corresponding plug-in loaded by daemon : flowvr::plugins::Scatter
Wait for receiving one message per input (nb inputs), and then send on output (use stamp read from message on in0)
Corresponding plug-in loaded by daemon : flowvr::plugins::SignalAnd
A shell around a filtersignaland component with one input port masking the actual number of ports of filtersignaland.
Forward on output the message received on input corresponding to the it stamp or received order.
Corresponding plug-in loaded by daemon : flowvr::plugins::FilterStamp
Forward input messages on outputs only if open (0 received on open port means closed)
Corresponding plug-in loaded by daemon : flowvr::plugins::Stop
Alternatively receive an message on each input and forward it on output
Corresponding plug-in loaded by daemon : flowvr::plugins::UnrotateFilter
Extra filters can be programmed if required.
A synchronizer is an other special type of filter that receives and send only stamp messages. It is dedicated to messages synchronisation.
It is instantiated using a syntax similar to filters by adding a <synchronizer> node and specifying the synchronizer to use in a <synchronizerclass> node.
FlowVR provides the following synchronizer (see flowvrd/src/plugins/sync for an up-to-date list)::
A synchronizer which chooses the last available message. Every time it receives a message on endIt port, it looks for the most recent message received on stamps port and sends to order port the stamp of this message. Used for implementing greedy (subsampling) filtering.
Corresponding plug-in loaded by daemon : flowvr::plugins::GreedySynchronizor
Send a first message and then send output messages at min (incoming message rate, freq rate)
Corresponding plug-in loaded by daemon : flowvr::plugins::MaxFrequencySynchronizor
A component must be developed for each module and metamodule. A module component reflects the interface of the module. A metamodule associates a launching command to a group of logically related modules.
The primes application defines 3 modules and 3 metamodules:
All modules are simple and similar. They inherit from the Module class. Everything takes place in the constructor:
See for instance share/flowvr/examples/primes/include/primes/components/modulecompute.comp.h
Metamodules are similars (they are components too). They inherit from the MetaModuleFlowvrRunSSH class. This specific class of metamodule is reserved for metamodules using the flowvr-run-ssh launching command (see ). Other metamodule class exists for different launching commands, and the user can easilly develop its own if required. This metamodule class takes as template the class name of the module to launch. In the constructor, the user must set the name of the executable (CmdLine("visu") for the visu metamodule). Because MetaModuleVisu opens a displays, it customizes the the launching command to open the display :0 calling:
getRun()->openDisplay(":0");
These metamodule does not explicitly define any port. They actually duplicate the ports of the module they encapsulate. For instance MetaModuleVisu has the same ports than ModuleVisu: "keysIn" and "primesIn".
MetaModuleFlowvrRunSSH assumes it can launch several instances of the module it contains. All details
Designing a complete application can be difficult for the FlowVR beginner. The use of network components and their placement in an application is not always easy. The goal of this section is to give some recipes about network design.
A connection between two modules links an output port to an input port. It is the responsibility of the FlowVR user to ensure that the type of data sent by the output port is compatible with the receiving module. In this case the connection between the modules is FIFO: the receiver module visu gets the message data in the order they were put by the emitter module compute. Each new call to a wait performed by visu blocks until a message is available on the port.
Cycles in connections are possible. In the Primes example, the compute module send calculated prime numbers inside small packets, in order to let the visu module build the spiral pattern progressively. To avoid the latter to be flooded with messages, a possibility demonstrated in the 1_fifo_one_to_one and 2_fifo_two_computes Primes test cases is to close the communications in a cycle. A connection relying on endIt/beginIt predefined ports is then added to let visu warn compute module is ready to receive new data. This creates a cycle. If the modules have been properly programmed, i.e putting a default value on each output port before the first call to the wait method, the modules will not deadlock when calling the wait.
Messages can easily be duplicated and sent along several connections (fan-out) using a routing node. For instance it is very useful for parallel computations where several routing nodes are used to build a broadcast along a binary tree. Each Message put by visu will be received by all compute modules in a FIFO order.
Filters are usually used to control the dataflow between modules. A filter can be used to send/discard chosen messages along a connection (keeping only the messages verifying some preconditions, like an iteration number, for instance), to gather data from different incoming connections, to create a new message, etc.
Often the beginner uses a (meta)module where a filter would have been more appropriate. Here are the key arguments to help you make the right choice:
Amongst filters provided with FlowVR, The merge and scatter filters are good examples of what is possible to do with filters.
The scatter filter enable to split a message in parts, each part being sent on a different output. The scatter is often necessary when using parallel modules. Scattering allows to reduce the bandwidth by sending only data used by the receiving modules.
Similarly, merging messages with a merge filter is sometimes required to built a message containing the complete result of a parallel metamodule where each module produces only a part of the result. The first example is a merge to gather the results of the compute metamodule of the prime example, while the second example uses a tree of merge filters to built the density grid computed for the fluid example.
|
The various test cases of the Primes example aim at demonstrating how network design takes a crucial part in the final behavior of a distributed application. We encourage you to run the different tests in order to feel the impact of each modification. Please refer to the share/flowvr/examples/primes/README for the launching commands.
This is the default mode when no synchronization object controls a connection. In this case, all messages produced by the emitter will be consumed in the same order by the receiver. No data will be lost or inverted in FIFO mode. Consequently all modules of the application will work at the speed of the slowest module. Moreover, if a module like a tracker works a lot quicker than his receiver, the size of the buffer holding the data between the modules will increase continuously, conducting to an overflow (crash) during execution.
In the Primes example, the tests 1_fifo_one_to_one and 2_fifo_two_computes uses a FIFO mode that makes the visu module work at the same speed as the slowest connected module, either compute or capture. To prevent messages overflow, the communication scheme is organized in a cycle.
The sampling (or greedy) mode refers to sampling systems where components periodically read data whose values are independently (asynchronously) updated (digital temperature sensor for instance).
This mode is designed for low latency and real time applications. This synchronization method allows modules to work at their maximum speed. At each new iteration, the module uses the latest data available, ignoring all precedent data. By this way, the latency can be improved but some data produced are lost. This mode can be implemented between two modules using a GreedySynchronizor synchronizer and a FilterIt filter.
The synchronizer has two input ports: stamps and endIt. stamps should be connected to the source of the data while endIt should be connected to the endIt activation port of the module receiving the data. With this information, the synchronizer will know when the module has finished an iteration and thus needs a new data. It will read the stamps received on the stamps port and find the most recent message. The chosen stamps will be forwarded to the order output port. This port should then be connected to the FilterIt filter on its order input port. The filter has another input port named in which should be connected to the same source of data that the synchronizer is. Using these informations, the filter will forward to its out output port the messages corresponding to the received orders. This output should then be connected to the input port of the destination module which will so receive the sampled data.
The Primes example introduces this technique in the test 4_capture_greedy, between the capture and visu modules. Run the test to observe the difference with the FIFO mode. The receiving module visu has its endIt port connected to the synchronizer GreedyKeys/sync/0. It enables this module to ask a new message to the synchronizer each time it completes an iteration. The synchronizer has a second input port to receive the stamps of each message sent on the output port of the source module capture. When the synchronizer receives a message from visu, it selects amongst the available stamps the most recent one and sends this stamp to the filter GreedyKeys/filter/0. If no new data has been received since the last iteration, the synchronizer replays the same stamp. The filter waits on its input port a message with a matching stamp and forward this message on its output port. This is the message that will receive the module visu for the pending iteration. Notice that thanks to the greedy, the initial cycle introduced with FIFO connections is suppressed.
The iteration frequency of a module can be bounded using a a MaxFrequencySynchronizor synchronizer. The synchronizer is simply connected to the beginIt input port of a module and set the synchronizer parameter the maximum authorized frequency.
In the Primes example, the visu module is constrained to a movie-like framerate of 25 Hz using this synchronizer.
By implementing other synchronizers and filters, it is possible to express other synchronization modes. In the same application, several synchronization modes upon different connections can be present.
In particular a "synchronized sampling" can be implemented by having a synchronizer controlling one or several filters. In addition to the traditional greedy, the Primes example exposes another synchronizer-filter pair in the 3_compute_mergeIt test. Instead of a FilterIt filter, the greedy variation assigned for a proper desynchronisation of the compute module is built with a MergeIt filter. This component has the ability to accumulate messages iteration by iteration, which is more suitable for messages delivering computed data that should not be lost. When the visu module request the information, it receives an all-in-one message. In the case visu is faster than compute module and no more new messages have arrived since the past request, an empty message is delivered. Thus the visu module runs at full speed. Refer to the GreedyItPrimes network component for an illustration of a FilterIt filter paired with a greedy synchronizer.
With both FilterIt and MergeIt synchronized connection for capture and compute modules, the Primes example as in the test 4_capture_greedy is fully greedy. The computation module is usually the slowest of the application. The greedy synchronization between computation and visualization allows the latter to display at a better frame rate independently of the speed of the computation. The second greedy connection is for minimizing the latency of the input device. For a better interaction, it is important for the application to use the last input status when starting a new iteration. It is important to keep in mind that if one of these connections are not greedy but FIFO, implicit synchronizations appear, and the greedy connection becomes useless. This is why it is generally recommended to have only FIFO or greedy synchronization on a specified FlowVR application.
Like modules, filters and synchronizers need to be mapped on a host. If this choice is not determinant to make the application work, it can have an incidence on the performance. Generally a filter or a synchronizer is placed either on the host running the source or destination module it is related to. The choice between these two solutions depends on whether the bandwidth or the latency should be favored.
To minimize the network traffic, it is better to put the filters and synchronizers on the host running the source module of the connection. It will avoid sending over the physical network messages that are not required by the destination module. To favor latency, it is better to put the filters and synchronizers on the host running the destination module of the connection. The interactions between the destination module and the filters and synchronizers will be more reactive as messages exchange will be just a pointer exchange through the shared memory and not an effective data transfer over the physical network.
Other intermediate solutions can be appropriate to take advantage of the specific network architecture of the cluster.
FlowVR requires a daemon on each host involved in a FlowVR application execution, including the host where the application is launched from.
A FlowVR daemon must run on each host running modules. The command line for launching a daemon on a host is:
flowvrd -top -net
The -net option is mandatory when the deamon is invovled in applications distributed on different machines. Each daemon can be launched in any order and not with the same user login shell (not recommended).
The -top option request the demaon to monitor and display some data about the plugins and modules it controls. In particular it gives the frequency of each running module or filter.
To obtain the list of available options for flowvrd :
flowvrd -h
See SharedMemorySegmentSharedMemorySegment for shared memory segment issues.
A FlowVR daemon is controlled by a XML based command language. See flowvr-parse/dtd/commands.dtd for the authorized syntax.
The commands for a given application are stored in te .cmd.xml file.
A FlowVR application developer only have to launch and stop daemons. He should not have to directly use the command language.
For the interested reader refer to the developer manual that contains more details about the daemon.
Use the -x option to request flowvr to start the application after processing.
For starting the Primes application:
flowvr -x Primes
Once the application is running, flowvr does not return and stay listening on the command line from user inputs suspend, stop and restart the application.
To supsend the execution:
pause
To resume the execution:
start
To stop the execution (all modules receive an exit signal through the wait call and all plugins (filters, synchronizers; etc.) the deamons may have loaded are unloaded:
stop
This command does not stop the daemons.
If the application does not start or stop correcly, it can results in zombie processes polluting the machines. The flowvr-kill utility enable to force killing all remaining processes.
Call flowvr-kill providing the list of hosts that have been involved in the execution:
flowvr-kill -list 'host1 host2 host3'
It is also possible to list the hosts in a file and provide the file to flowvr-kill. You only have to list the hosts, one per line, and save the file as txt.
flowvr-kill -hostfile mycluster.txt
When a module calls the init() method, it creates a temporary file locally in the /.flowvr directory. This files stores the PID of the module and the name of the machine. When flowvr-kill is called it just inspect on each machine the /.flowvr directory and kill all listed processes.
2009-04-06
