MIF is a Java API for constructing analytical and scientific processing pipelines which runs on top of Mule. MIF runs in distributed environments on any Java-supported platform and integrates codes written in other languages. MIF also contains graphical tools to help application programmers and end users rapidly build high-quality distributed applications.
Mule is incredibly flexible, powerful, and fast. We started using Mule's XML configuration capabilities to do analytic integration around 2005 and developed several successful projects. However, our user base of application programmers was not able to get up to speed quickly enough during short term projects to create applications on their own. We had to do extensive hand-holding by writing the XML configurations for them while describing the programming model they should follow. This method of integration proved to be inefficient, so we created a simplified programming model around the Mule API. Immediately we found that application programmers were able to get up to speed more quickly because many of the details of creating a Mule pipeline were hidden from them. Since then, we have increased the scope of the programming model and added several features to incorporate data provenance capture, data caching, workflow integration, remote execution, integration of code written in other languages, and more.
Component Builder: We are currently building a graphical designer based on Eclipse model-driven development technologies. This tool will allow an application programmer to automatically generate most of the code needed to create a MIF pipeline.
Component Repository: Components created using the Component Builder are stored in an RDF database for later use by a graphical workflow designer.
… And more on the way!
- A distributed cyber-security application deployed in production at a major conference
- Integration of bio-informatics tools into an analytic application
- A distributed fusion and triage architecture for analysis of simulated shipment and cargo data
- High speed video processing in a security context
- Data stream filtering and processing for situational awareness
- Sensor network processing and analysis
For further guidance on the range of applications that MIF has been used in, please see the MIF Architectural Overview. If you are a technical user who is interested in programming with our tool chain, browse through the MIF Programmer Guide. To get an idea of the academic background behind MIF, see our Papers section.
How long is a piece of string?
Since MIF is based on Mule, we inherit most of its performance characteristics. In general, we have found Mule to be extremely lightweight and our overriding goal is to introduce as little additional overhead as possible. We have done performance testing which indicate data transfer time between components for 1K messages of 100 microseconds and 256K messages of less than 2 milliseconds. We have also been able to push 4000 messages/second through a simple pipeline and scale applications by distributing Mule across multiple machines. We have created a benchmark suite which is available in the MIF distribution.
Of course, efficient applications require extensive tuning of the virtual machine, thread pools, message transports, and application algorithms.