Skip to Main Content U.S. Department of Energy
 

MIF Hello Ingester Sample

The helloIngester sample demonstrates the MifIngesterModule, which is a specialized module that feeds data as a series of messages into a MIF pipeline. An interface is provided by MIF that allows the programmer to specify how to get the next data record, which could be the next line in a file or the next record returned from a database query.

Sample Overview

In this sample, an ingester called HelloIngesterProcessor reads a series of names from an array of strings. Each name is sent as a separate message to the next module in the pipeline, which is implemented by the HelloHalProcessor used in other samples. The ingester works by simply keeping an index into the array so that it can keep track of where it left off. Then, MIF calls the processor's getNextMessage() method which returns the next name and increments its current index.

This is a simplified example. A more typical scenario, pictured below, is an ingester which reads its data from a file, and sends each line as a separate message into the MIF pipeline. Actually, the programmer can implement whatever logic is desired into the getNextMessage() method as long as it gets the next item in the dataset and sends null when it has completed.

Hello Ingester

Running the Sample

This sample consists of a single MifPipeline, which prompts you to press enter to start the ingester. The ingester module then sends a series of strings as separate messages to another module, which prints the string along with some additional text to the console.

From the command line

Navigate to MIF_HOME and run the command:

bin/mif.sh -c gov.pnnl.mif.samples.hello.ingester.HelloIngesterDriver

MIF will have started when you receive the message:

2010-05-27 11:01:37,326 INFO  [MifManagerImpl]  -- MIF has started --

The console will then prompt you to press enter to start with the message:

Hit enter to start.

After pressing enter, the resulting display should be something similar to the following:

HelloIngester: getting next message from array --> ed
HelloHalProcessor: ed, what are you doing?
HelloIngester: getting next message from array --> joe
HelloHalProcessor: joe, what are you doing?
HelloIngester: getting next message from array --> hal
HelloHalProcessor: hal, what are you doing?
HelloIngester: getting next message from array --> steve
HelloHalProcessor: steve, what are you doing?
HelloIngester: getting next message from array --> mark
HelloHalProcessor: mark, what are you doing?
Ingester: No more data. Returning null.

In this output, HelloIngester announces on the console that it is ready to send the next name, which it sends to the HelloHalProcessor, which greets that person with a message on the console. The HelloIngester then annouces that it is done by returning null after it has sent all the names from the array. Any subsequentattempts to hit enter will result in the ingester stating that it is done sending names:

No more data. Returning null.

Code Walkthrough

All the source files for this example can be found in MIF_HOME/sources/mif-samples/src/gov/pnnl/mif/samples/hello/ingester.

HelloIngesterProcessor

The HelloIngesterProcessor class implements the MifIngesterProcessor interface, which provides the getNextMessage() method for the programmer to implement. It is up to the programmer to impelement any logic necessary to get the next item in the dataset. As mentioned above, this could be a file in which each record is a line. In this case, we simply have a fixed string array and maintain an index into the array so that we can get the next one when MIF calls getNextMessage().

public class HelloIngesterProcessor implements MifIngesterProcessor {
 
public Serializable getNextMessage(Serializable pipelineData) {
  String msg;
 
  //while there is more data to ingest, return the next message
  if (counter < testNames.length) {
    msg = testNames[counter++];
    log.info("getNextMessage -- " + msg);
  }
 
  //when the data runs out, return a NULL to mark that hte data is done being ingested
  else {
    msg = null;
    log.info("No more data. Returning null.")
  }
 return msg;
 }
}

Setting Up the Pipeline (main)

The pipeline that uses the above IngesterProcessor is created by instantiating a MifPipeline object as demonstrated in previous samples. Then, the MifIngesterModule is created by giving it a stdio inbound endpoint and a vm endpoint to send data to the next step:

pipeline.AddmIfIngesterModule(HelloIngesterProcessor.class,"stdio://stdin?promptMessage=Hit enter to start.", "vm://queue-1");

We then add a MifModule to the pipeline, which simply greets the person whose name it receives

pipeline.addMifModule(HelloHalProcessor.class.getname(), "vm://queue-1", null);
 
hello_ingester.txt · Last modified: 2010/05/27 11:22 by adamw