Skip to Main Content U.S. Department of Energy

Specialized Module Configuration

Specialized modules are MifModules that were designed for a specific purpose and have implementations that are provided by MIF. Thus, no MifProcessor class needs to be implemented when using these modules. This page documents properties that can be set on these modules to control and customize their behavior.

MifIngesterModule

The MifIngesterModule can “ingest” data into a MIF pipeline by reading data from the filesystem, database, etc., and sending individual data records as messages into the MIF pipeline.

Properties

property values default description
sendSynchronous boolean true Whether the module sends synchronously to the pipeline

BridgeModule

The BridgeModule simply receives data on its inbound endpoint(s) and sends out its outbound endpoint(s) without performing any processing. This is useful for testing and in cases where an aggregator is needed but there is no associated processing to do.

Properties

property values default description
verbose boolean false If set to true, the module prints all received messages to the console

DataMoverModule

This module pulls data on disk from a source location (sourceFileUri) and puts it into a destination (destinationFileUri). The data transfer is triggered by event messages from other modules. See the DataMoverModule sample for more information.

Properties

property values default description
sourceFileUri string null This required property represents the URI where the files should be retrieved from.
destinationFileUri string null This required property represents the URI where the files should be deposited.
sshDirectory string null Required for ssh URIs. Used to locate the local user's passphraseless ssh-key

Supported protocols

These are the supported protocols and example URIs for the sourceFileUri and destinationFileUri properties listed above.

protocol example description
file file:///tmp/file42.txt Use the standard file URI scheme syntax
scp scp://user@host/path/to/file42.txt This implies passphraseless ssh keys are set up on your system. Must also set sshDirectory property
http http://localhost:9090/service Valid for source URI only
dynamic dynamic://message.property.key A URI that can be dynamically resolved at runtime based on a property in the message header. Path of URI is the key of the message property where the runtime URI is stored. Only valid for destination URIs
bbcp bbcp://user@host/path/to/file42.txt Not yet fully implemented

ExternalCommandModule

This module will launch an external OS command, either on the local machine or on a remote machine. If it is a remote command, it is executed by logging into the destination machine via ssh. In this case, a passphraseless ssh key is assumed to be present, which is specified with the sshDirectory property.

Properties

property values default description
sshDirectory string null Path to the local user's directory where a the ssh keys are located
returnCommandResult boolean true If true, send the result of the command (ie, return value) via the outbound endpoint. If false, send the console output via the outbound endpoint
 
specialized_module_configuration.txt · Last modified: 2010/06/08 09:29 by adamw