Data-Intensive Computing

I am currently working on the distributed execution of flows as part of the Meandre infrastructure—as a part of the SEASR project. One of the pieces to explore is how to push data between machines. No, I am not going to talk about network protocols and the like here, but how you can pass the data around. If you have ever programmed MPI using C/C++ you remember the tedious efforts that requires passing complex data structures around between processes. Serialization is a way to take those complex structures into a form that can be easily stored/transmitted, and then retrieved/received and regenerate the original complex data structure. Some languages/platforms support this functionality (e.g. Java, Python), allowing to easily use the serialized representation for persistency or transmission purposes. ...

by Llorà, X. IlliGAL technical report 2009001. You can download the pdf here. Abstract: Data-intensive computing has positioned itself as a valuable programming paradigm to efficiently approach problems requiring processing very large volumes of data. This paper presents a pilot study about how to apply the data-intensive computing paradigm to evolutionary computation algorithms. Two representative cases—selectorecombinative genetic algorithms and estimation of distribution algorithms—are presented, analyzed, discussed. This study shows that equivalent data-intensive computing evolutionary computation algorithms can be easily developed, providing robust and scalable algorithms for the multicore-computing era. Experimental results show how such algorithms scale with the number of available cores without further modification. ...

Today, Meandre 1.4.0 has been released. The final release includes the infrastructure, tools (Workbench, ZigZag, etc.), and a set of component and flow repositories. Also, 1.4.1 will be also released later today adding the latest feature updates and few bug fixes on the tools.

Today I just tagged the final release candidate RC6 for the Meandre Infrastructure. The list of improvement from the previous 1.3.2 release is long, but I just want to highlight a few of them. Standardization of web services and their response information. Also added new services to help manage users, repositories, and jobs. The infrastructure is now able to provide a single system image cluster of Meandre servers. It also provides a new light weight HTML interface for easy administration. Location addition has been speeded up minimizing the network usage. Component contexts are locally cached, drastically reducing the preparation time of a flow before executing. Also, all the reported bugs reported so far has been fixed. Hopefully, the release will be pushed out this coming week. Stay tuned :D. Also, a few documentation for developers interested in the infrastructure can be find on the development tools Confluence wiki. ...

Last monday I tagged on the development SVN the first release candidate for the 1.4 version of the Meandre infrastructure. The new 1.4 version brings a pile of bug fixes, improves web service mechanics, provides a new HTML lightweight interface to the infrastructure, and implements a first version of the coordination facility (Meandre Distributed eXchange) used to assemble configurable single-image Meandre clusters based on Meandre servers—just to list a few of the 1.4 goodies. So far there is no firm release date, but it will definitely be before the SEASR 2009 Workshop on January 15th, 2009. ...